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INTRODUCTION 

There is growing public recognition that adequate levels of scientific literacy 
are not being attained by many children and adults (American Association for 
the Advancement of Science, 1989; Science Council of Canada, 1984). 
Because the quality of life in the 21st century depends upon on an educated 
populace that is scientifically and technologically competent, the problem of 
scientific literacy deserves serious attention. This manual was designed for 
use in a research project focused on one facet of scientific literacy, namely, 
how individuals assess the credibility of brief reports of treatment studies. 
These reports are pervasive in classroom textbooks and the popular media. 
Our interest is in (a) the types of requests for information that people make 
about these reports as an index of their potential to assess credibility of 
research conclusions and (b) reasons people have for asking these questions 
as a means to make inferences about the knowledge structures that give rise 
to their requests. 

Questions are the basis of scientific investigation. They are a hallmark of 
good thinking and a good education. They are fundamental to the 
instructional dialogue in any classroom and the informal types of assessment 
that teachers make about what students know. Empirically, we hope that an 
examination of peoples' requests and their justifications for the requests will 
allow us to determine (a) what people know about features of scientific 
research, (b) how, when, and whether people engage in evaluative thinking 
about these features when reading reports of scientific investigations, and (c) 
how these characteristics of knowledge and thinking vary as a function of 
age and schooling from adolescence through early adulthood. Our research 
is conducted in the information processing tradition in psychology, but our 
theoretical agenda is also structuralist and contextualist in flavor: We seek 
to use these responses to make inferences about the underlying knowledge 
structures that are involved in the evaluation of scientific research, as well 
as to understand how structures are related to opportunites to learn, including 
classroom experiences, general educational background, and disciplinary 
training. This manual may be of value to anyone with similar objectives. It 
can be used to characterize the performance of groups as well as individuals. 

In writing this manual, we had to make two assumptions about the 
background of the individuals most likely to use it. First, coders should be 
knowledgeable about science and the process of conducting science--at least 
as knowledgeable as the individuals whose responses are being coded. 
Second, the coder should have the equivalent ot a 4-year university degree in 
the sciences, or social sciences, and have some knowledge of probability and 
statistics to adequately categorize people's responses. Others may be able 
to use this manual, but special, more elaborate explanations of concepts and 
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terminology may be required. 

Participants' Task 

The series of studies designed to examine scientific literacy in this 
project required that the participants complete a questionnaire containing 
several news briefs. Participants were asked to read each news brief 
carefully and then list the information they would need or questions they 
would want answered to help them evaluate the conclusion of the news 
brief (e.g., "What additional pieces of information would you like to have 
about the researchers' report to decide whether the conclusion. ..is 
correct?" or "Please generate a list of as many questions as you can 
that you would want to have answered before you decide whether the 
conclusion. ..is true. What is the first question that you would want 
answered?" then, "What is the next question you would want 
answered?"). In some studies, participants were also asked to justify 
how this information would help to evaluate the conclusion (e.g., "How 
would an answer to this question help you to decide whether 
the. ..conclusion in the news brief is true?" or "Why would you would 
want each question answered?"). Samples of the news briefs and the 
questions used to prompt the participant are provided in Appendix 1, 

Coding Task 

The information that participant sp.id they would need or the questions 
they posed will be referred to as 'requests'. Requests are generally 
'who', 'what', 'where', 'why' and 'how' questions or phrases that can 
be turned into these types of questions. 

Responses that appear after the prompts, "How would an answer to this 
question help you to decide whether the underlined conclusion in the 
news brief is true?" or "Why you would want each question answered?" 
will be referred to as justifications . Justifications are generally 
statements that could begin with the word because . These 
statements provide the reason or motivation for the request and reflect 
the participants' knowledge or understanding of the importance of the 
information requested. 

When coding responses, there are two primary goals. First, the coder 
must identify the topic of the request. The topic can be thought of as 
the basic theme or idea underlying the request. Descriptions of the 
topics to be used when coding are provided in the section "Topic 
Descriptions". Second, the coder must identify the domain of knowledge 
that is addressed by the justification. The knowledge domain is the core 
knowledge about science that prompts the participant to request the 
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information that they see as important to evaluating scientific research. 
Descriptions of the knowledge domains to be used when coding are 
provided in the section "Knowledge Domain Descriptions". 

General Procedure for Coding 
The procedure for coding is: 

1) Read the response to the first prompt, which should be a 

request for information. 

2) Identify the main idea in the request and assign a corresponding 

topic code. 

3) Read the response to the second prompt, keeping in mind that 

this response is a justification for the initial request. 

4) Identify the central idea that constitutes the justification for the 

request and assign a corresponding knowledge domain code. 

Ideally, this means that participants should express a single idea in 
response to the first prompt which can be given a topic code and then 
provide a single justification for the request which can be given a 
knowledge domain code. However, participants do not always do things in 
a straight forward manner. They may identify more than one idea in 
their initial request for information, the justification may not be related to 
the initial request, the justification may be stated as part of the initial 
request, or no justification may be provided. Detailed procedures for 
dealing with these instances are outlined throughout the manual. 

Coding Principles 

Use the following principles to assist with coding: 

PRINCIPLE 1 : Assume that the participant is doing the task. 

When assigning topic codes, assume that ideas following prompts such 
as "What is the next question you would want answered?" are requests 
tha* receive topic codes. 

When assigning knowledge domain codes, assume that ideas following 
prompts such as "How would an answer to this question help you to 
decide whether the underlined conclusion in the news brief is true?" are 
justifications for the requests that should receive knowledge domain 
codes. 



Finally, assume that particioants use each numbered space in the 
questionnaire to reflect a single idea that can be captured in the 
taxonomies either as requests or as justifications. 
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This principle has implications for (a) assigning codes for topics and 
knowledge domains (b) identifying the link between the requests and the 
corresponding justifications, and (c) partitioning responses into units 
that represent separate requests and justifications. The implications of 
these principles for coding are detailed in the sections entitled 
"Assignment of Topics", "Assignment of Knowledge Domains", and 
"Identification of Requests and Justifications", respectively. 

PRINCIPLE 2: Topic codes should be assigned based on information in 
the requests only, whereas knowledge domain codes should be made 
based on the information in both the justification and related request. 

When classifying ideas that are requests, use of information from the 
justification portion of a student's response is prohibited . Use of this 
information can undermine inter-rater reliability and affect the validity of 
comparisons between studies. When classifying ideas that are 
justifications, use of the request information is essential . To give a 
knowledge domain coding, the coder should ask, "How does the 
justification explain (or motivate) the request?" 

PRINCIPLE 3: There are only three classes of requests and justifications. 
Responses may be (a) rudimentary or ambiguous with respect to the 
categories listed in each taxonomy; (b) representative of one of the 
categories listed in each taxonomy; or (c) representative of an "other" 
category. 

Rudimentary ideas are the most naive type of response. They address 
an idea in a simple superficial way. (e.g., What methods did they use?). 
Ambiguous ideas are vague responses. They seem to be loosely related 
to more than one category, but there is not enough information in the 
responses to assign them to a single category. 

Responses that a.e coded as other, address a specific idea that is not 
reflected in the categories. 

Therefore, to assign either topic or knowledge domain code, the coder 
should identify the top level category that best fits the request or 
justification and then ask three key questions: 

a) Is this response rudimentary or ambiguous with respect to the 

categories listed? 

b) Does this response fit one of the categories listed? 

c) Does this response belong to another category not specified 

in the toxonomy? 
Every request or justification belongs to one of these three classes. 
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PRINCIPLE 4: Codes for justifications can be assigned to 
know/edge domains on the basis of a strict or lenient criterion. 

A lenient criterion is being used when a inference is required to assign a 
knowledge domain code to the justification. In this case, the reason or 
motivation is not clearly specified in the participant's response, but the 
justification seems to be related to a knowledge domain in the 
taxonomy. To be clear of the reason or motivation behind the 
participant's request, the coder would have to continue to ask the 
participant why this information is important. A strict criterion is being 
used when a knowledge domain code is assigned to a justification only 
when the reason or motivation for the request is clearly spelled out. 
Participants often only hint at the reason or motivation they have for 
making a request, so a more lenient criteria is typically used to capture 
the majority of the data. 

The procedures and guidelines for coding requests and justifications 
according to these principles are outlined in the sections that follow. B; ( ef 
references to these principles are made throughout the sections on 
"Assignment of Topics" (below) and "Assignment of Knowledge Domains" 
(p. 50). Use these descriptions to clarify the coding procedure as needed. 

ASSIGNMENT OF TOPICS 

This section is focused on the procedure for assigning topic codes which 
characterize the ideas that appear after prompts like: "What additional pieces 
of information would you like to have about the researchers' report to decide 
whether the conclusion ... is correct?" or "Please generate a list of as many 
questions as you can that you would want to have answered before you 
decide whether the conclusion ... is true." 

Topic assignment means identifying the gist of the request. Topics reflect 
the content of the request and topic codes should reflect the main idea of the 
request. The topic is the focus of the request. Requests may vary in 
specificity, length or complexity. 

When assigning a topic code, it is important to take each request as it is 
written and not add information to the request that may change it. Ideally, 
the information requested should be related to the research in the news brief. 
If no logical link between the request and the news brief can be found then, 
the participant is not doing the task and the request should be coded as "off- 
task". 
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STEPS FOR ASSIGNING A TOPIC CODE 

REMEMBER PRINCIPLE 1; Assume the participant is doing the task. 

REMEMBER PRINCIPLE 2: Topic codes should be assigned based on 
information in the requests only. 

1 ) Read the response that appears after prompts like: "What is the 
first/next question you wouid want answered?". 

2) Decide what the request is about. To do this it may be necessary to 
paraphrase the response to the first prompt. Be especially careful when 
paraphrasing not to add anything to the response that was not part of 
the initial statement. 

3) To assign a topic code, decide which of the 'top level' categories 
encompasses the request. That is decide whether it is related to Social 
Context (1.0), Agent (2.0), Methods (3.0), Data/Statistics (4.0), 
Relevance (5.0), Related to Other Research (6.0), is some other topic 
(7.0), is ambiguous but relevant (8.0), or is completely off-task (9.0). 

REMEMBER PRINCIPLE 3: There are only three classes of requests. 
Requests may be (a) rudimentary or ambiguous with respect to the 
categories listed; (b) representative of one of the categories listed; or (c) 
representative of an "other" category. 

4) Once the general area is identified, examine the subcategories within 
that section to determine whether or not the request can be classified 
further. Use the most specific level of classification that the request 
clearly fits into . 

a) Recall, from the description of Principle 3, that requests that are 
too rudimentary or are too ambiguous to be given a specific 
subcategory topic code should be given a topic code that reflects a 
more general level of classification. Note that if the request is too 
rudimentary or ambiguous with respect to the top-level categories, 
the request should be coded as 8.0 Ambiguous but Relevant. 

b) If requests reflect information that is not captured by the specified 
categories, code the request as an "OTHER" category in line with 
the top-level category that best describes it. Note that if there is no 
top-level category that reflects the gist of the request then the 
request should be coded as 7.0, Other. 
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c) If the request clearly addresses more than one specific topic, it 
will be necessary to separate the request into separate ideas 
according to the guidelines set out in the section on "Identification 
of Requests and Justifications", and code each idea separately. 

NOTE: It may be difficult to decide whether the request reflects two 
separate topics or is an ambiguous request. If you can code the 
request as topic 'a ' AND topic 'b ', then it reflects two ideas; if you 
can code the request as topic 'a ' QR topic 'b ', then it is ambiguous. 

NOTE: Examples provided with each topic description in the next 
section are not exhaustive and requests may represent the same 
idea in different ways. Examples were taken from student protocols 
(indicated with by P) or generated by the authors to facilitate 
coding. A summary of topic codes is provided in Appendix 2. 
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TOPIC DESCRIPTIONS 

1 .0 SOCIAL CONTEXT 

Requests included under this topic indicate an awareness of social factors 
which may influence the quality of the research and the validity of the data or 
conclusion. Questions included under Social Context may refer to the 
researchers or experimenters who conducted the study described in the news 
brief, promoters/detractors of the results reported in the news brief (e.g., Mind 
Matters), creators of the agent (i.e., who made the agent Ollinite), experts 
(e.g., other researchers/experimenters) and other persons who may be 
associated with the research (e.g., have an opinion on the research). These 
topics are found under section 1.1 People. Also included under Social Context 
are the topics concerning the publication outlet of the research and the 
author(s)/broadcasters of the publication. These topics are found under the 
section 1.2 Source of Information. Finally, included under Social Context are 
topics concerning other elements of the research context inc'-ding topics 1.3 
Funding Issues, 1.4 Identification of Research Institution, and 1.5 Source of 
the Research Question. Questions that refer to a specific and identifiable topic 
related to social context, but are different from topics 1.1 to 1.5 are 
categorized under 1.6 Social Context - Other. 

Requests that refer to topics concerning Social Context but are too 
rudimentary or ambiguous to be categorized under topics 1. 1 to 1.6 (i.e., 
refers to more than one topic listed under Social Context) are categorized as 

1.0 Social Context. 

Example 

Is this a marketing scam? (P) 

1.1 PEOPLE 

Included under this section are topics concerning the researchers or 
experimenters of the study described in the news brief (1.1.1). the 
promoters or detractors of the results described in the news brief (1.1.2), 
the creators of tie agent (1.1.3), and other experts who may have an 
opinion on the research (1.1.4). Specific and identifiable questions that 
refer to people but are different from topics 1.1.1 to 1 .1 .4 are categorized 
as topic 1.1.5 (People - Other). 

Requests that refer to People but are too rudimentary or ambiguous to be 
categorized under topics 1. 1. 1 to 1. 1.5 (i.e., the respondent confuses the 
researchers with the promoters) are categorized as topic 1. 1 (People). 
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1.1.1 Researchers/Experimenters 

This topic concerns individuals who conducted the research described 
in the news brief. Included under this topic are questions about the 
identification of the researchers/experimenters (1.1.1.1), the number 
of researchers/experiments involved in the study (1.1.1.2), their 
testimonials regarding the research (1.1.1.3), 
training/qualifications/ci'edentials of the researchers/experimenters 

(1.1.1.4) and motivations/biases of the researchers/experimenters 

(1.1.1.5) . Specific and identifiable questions that refer to the 
researchers or experimenters, but are different from topics 1.1.1.1 to 

1.1.1.5 are categorized under Researcher/Experimenter - Other 

(1.1.1.6) . 

Requests that refer to the researchers or experimenters but are too 
rudimentary or ambiguous to be categorized under topics 1. 1. 1. 1 to 

1. 1. 1.6 (i.e., fit in more that one topic listed under 
Researcher/Experimenter) are categorized as topic 1.1.1 
f Researchers/Experimen ters). 

1.1.1.1 Identification 

This topic includes requests pertaining to the identity of the 
people who conducted the research described in the news 
brief. These questions are general (i.e., "Who were the 
researchers or experimenters?"). Detailed questions about the 
researchers' or experimenters' qualifications, motivations etc. 
are not included under this topic (see topics 1.1.1.4 Training, 
Qualifications, Credentials and 1.1.1.5 Motivation, 
Expectations, Biases, Beliefs). 

Examples 

Who are the researchers? 

Who are the people who conducted the research? (P) 

1.1.1.2 Number 

This topic includes requests pertaining to the number of 
researchers/experimenters involved in the research described in 
the news brief. 

Example 

How many researchers were there? 

1.1.1.3 Testimonials 
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This topic includes requests pertaining to comments or 
opinions that the researchers/experimenters may have about 
the research described in the news brief. 

Example 

What do the researchers have to say about Permaldrin? 



1 .1 .1 .4 Training/Qualification/Credentials 

This topic includes requests pertaining to the training, 
qualifications, credentials of the researchers/experimenters who 
conducted the research described in the news brief. 

Examples 

Are the researchers qualified to do this research? 
Where did the researchers get their training? 
How much experience do the researchers have? 

1.1.1.5 Motivation/Expectations/Biases/Beliefs 

This topic includes requests pertaining to the 
motivations, biases, or beliefs that the researchers or 
experimenters may have had when designing the study, 
conducting the research, and analyzing and interpreting data. 
The questions must clearly refer to the researchers' or 
experimenters' motivations, expectations, biases, or beliefs. 
For example, the question "Were the researchers paid to 'find' 
these particular . 2sults?" would be categorized under 1.1.1.5 
(Motivation/Expectations/Biases/ Beliefs). However, the 
question "Who funded the researchers?" would be categorized 
under topic 1.3 (Funding Issues). 

Examples 

Why did the researchers conduct this study? 

Did the researchers want the results to turn out in this 

particular way? 

Do the researchers believe the crystal will help people dream 
about the future? 

Why would the researchers want to study crystals? 
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1.1.1.6 Researcher/Experimenter - Other 

This topic includes identifiable and specific questions about the 
researchers/experimenters involved in the research described in 
the news brief but are different from topics 1.1.1.1 to 1.1.1.5. 



1.1.2 Promoters and Detractors 

This topic includes requests that concern the individuals who 
promoted the research that is described in the news brief (promoters) 
and\or individuals who defame the research that is described in the 
news brief (detractors). One group of promoters is introduced in each 
news brief (e.g., Mind Matters, Nature Unlimited, Autos for the 
Future). Also included under this topic are questions that refer to 
other potential promoters of the research but who are not introduced 
in the news brief (e.g., "What does Greenpeace have to say about 
this research?" is coded as topic 1.1.2.3). Finally, questions that 
refer to potential detractors of the research (not introduced in the 
news brief) are also included under this topic. The specific topics that 
fall under 1.1.2 Promoters or Detractors may refer to the 
identification (1.1.2.1), number (1.1.2.2), testimonials (1.1.2.3), 
training/credentials (1.1.2.4), and motivation/expectations/biases/ 
beliefs (1.1.2.5). Specific and identifiable questions that refer to the 
promoters or detractors of the research but are different from topics 
1.1.1.1 to 1 .1 .1 .5 are categorized as 1 .1 .1 .6 (Promoters and 
Detractors - Other) 

Requests that refer to the promoters or detractors but are too 
rudimentary or ambiguous to be categorized under topics 1. 1.2. 1 to 
1.1.2.6 (i.e., fit in more than one topic listed under the topic 
Promoters and Detractors) are categorized as topic 1. 1.2 
(Promoters and Detractors). 

Example 

Where did Mind Matters get their facts? 



1.1.2.1 Identification 

This topic includes requests pertaining to the identity of the 
promoters/detractors of the results of the research described in 
the news brief. These questions are geneial (i.e., Who are Mind 
Matters?). Detailed questions about the promoters' 
qualifications, motivations, etc. are not included under this topic 
(see topics 1.1.2.4 Training, Qualifications, Credentials and 
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1.1.2.5 Motivations, Expectations, Biases, Beliefs ). 
Example 

Who are Mind Matters? 



1.1.2.2 Number 

This topic includes requests pertaining to the number of 
promoters/detractors of the results described in the news brief. 

Example 

How many members are there in Mind Matters? 

1 .1 .2.3 Testimonials 

This topic includes requests pertaining to any comments 
or opinions that the promoters or detractors may have about the 
treatment above and beyond their endorsement described in the 
news brief. 

Examples 

What else does Mind Matters have to say about Ollinite? 
What does Greenpeace think about Permaldrin? 



1 .1 .2.4 Training/Qualifications/Credentials 

This topic includes requests pertaining to the training, 
qualifications, credentials of the promoters and detractors in 
regard to the treatment described in the news brief. 

Examples 

Are the members of Mind Matters qualified to make this 
statement? 

Are Members of Lifestyles for Seniors familiar with the process 
of meditation? (P) 

What kind of authority does Mind Matters have to make this 
statement? (P) 

How long have Mind Matters been an established organization? 
(P) 
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1.1 .2.5 Motivation/Expectations/Biases/Beliefs 

This topic includes requests pertaining to the motivations, 
expectations, biases, and beliefs that the promoters or 
detractors may have for promoting or refuting the results. 

Examples 

What do members of Mind Matters have to gain by promoting 
these results? 

Why did Mind Matters promote this research? 

How objectively did members of Nature Unlimited evaluate the 

researchers' findings? (P) 



1.1.2.6 Promoters and Detractors - Other 

This topic includes specific and identifiable questions about the 
promoters and detractors of the results described in the news 
brief but are different from topics 1.1.2.1 to 1.1.2.5. 



1.1.3 Creators of Agent 

This topic concerns the people or group of people who discovered or 
developed the agent. Included under this topic are questions that refer 
to Identification (1.1.3.1), Number (1.1.3.2), Testimonials (1.1.3.3), 
Training/Qualifications/Credentiais (1.1.3.4) and Motivation/ 
Expectations/Biases/Beliefs (1.1.3.5). Specific t id identifiable 
questions that refer to the creators of the agent but are different from 
topics 1.1.3.1 to 1.1.3.5 are categorized as Creators of Agent - Other 
(1.1.3.6) 

Requests that refer to the creators of the agent hut are too 
rudimentary or ambiguous to be categorized under topics 1. 1.3. 1 to 
1. 1.3.6 (i.e., fit under more than one topic listed under Creators of 
Agent) are categorized as topic 1. 1. 1 (Creators of Agent). 



1.1.3.1 Identification 

This topic includes general questions pertaining to the identity of 
the creators of the treatment agent described in the news brief. 
Detailed questions about the creators' qualifications, motivations, 
etc. are not included under this topic (see topics 1.1.3.4. and 
1.1.3.5). 

Examples 



!6 



Scientific Literacy 



15 

Who created this pesticide? 

Who discovered this crystal? 

Who developed this meditation technique? 



1.1.3.2 Number 

This topic includes requests pertaining to the number of 
creators of the treatment agent described in the news brief. 

Example 

How many people developed this technique? 



1 .1 .3.3 Testimonials 

This topic includes requests pertaining to comments or opinions 
that the creators of the treatment agent may have about the 
treatment agent described in the news brief. 

Example 

What do the creators of Quipmanol have to say about this 
conclusion? 



1 .1 .3.4 Training/Qualification/Credentials 

This topic includes requests pertaining to the training, 
qualifications, and credentials of the creators of the treatment 
agent. 

Example 

What kind of training did the creators of Permaldrin have? 



1 .1 .3.5 Motivation/Expectations/Biases/Beliefs 

This topic includes requests pertaining to the personal 
motivations or biases that the creators may have had for 
creating the treatment agent. These questions clearly refer to the 
creators' motivations and biases. Questions that refer to the 
pragmatic goal of creating the agent (e.g., "What is the 
Permaldrin used for?") are not included under this topic but are 
included under topic 2.2 (Agent Mechanisms and Effects/Side 
Effects). 
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Examples 

What goal did the makers of Quipmanol have in mind when 
creating this product? 

What would the creators of Permaldrin have to gain by allowing 
this situation to continue? 



1.1.3.6 Creators of Agent - Other 

This topic includes requests about the creators of the treatment 
agent used in the research described in the news brief but are 
different from topics 1.1.3.1 to 1.1.3.5. 



1.1.4 Other Experts 

ThL; topic concerns other researchers/experimenters or professionals 
in the area of research (other than the promoters and detractors or 
the creators) who may have an opinion on the research agent. 
Included under this topic are questions that refer to Identification 
(1.1.4.1), Number (1.1.4.2), Testimonials (1.1.4.3), Training/ 
Qualifications/Credentials (1.1.4.4), and Motivation/Biases 
(1 .1 .4.5). Specific and identifiable questions that refer to other 
experts but are different from topics 1 . 1 .4. 1 to 1 . 1 .4.5 are 
categorized as Other Experts - Other (1.1.4.6) 

Requests thot refer to other experts but are too rudimentary or 
ambiguous to be categorized under topics 1. 1.4. 1 to 1. 1.4.6 (i.e., fits 
in more than one topic listed under Other Experts) are categorized 
as topic 1. 1.4 (Other Experts). 



1 .1 .4.1 Identification 

This topic includes requests pertaining to the identity of other 
experts who may have an opinion about the research described 
in the news brief. These are general questions; they do not refer 
to the qualifications, motivations, etc. of the other experts. 
Detailed questions about the other experts' qualifications, 
motivations, etc. are not included under this topic (see topic 
1.1.4.4. and 1.1.4.5). 

Example 

What other proft: sionals are involved in this research? 
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1.1.4.2 Number 

This topic includes requests pertaining to the number of other 
experts. 

Example 

How many other qualified people are involved in the research? 



1 .1 .4.3 Testimonials 

This topic includes requests pertaining to comments or opinions 
that other experts may have about the research described in the 
news brief. This topic does not include questions about whether 
there was consensus about the conclusions reached by the 
researchers described in the news brief. Questions concerning 
consensus (e.g., "Do other researchers agree with this 
conclusion?") are categorized under topic 6.4. 
(Consensus/Nonconsensus). 

Examples 

Do other researchers have something to say about this? 
What do other researchers have to say about this? 
What do teachers have to say about this? 



1 .1 .4.4 Training/Qualification/Credentials 

This topic includes requests about the training, qualifications, 
and credentials of the other experts in the research area. 

Example 

Are other professionals trained in this area? 



1 .1 .4.5 Motivation/Expectations/Biases/Bei.jfs 

This topic includes requests about the motivations, expectations, 
biases and/or beliefs of other experts that may affect their 
opinions or comments about the research described in the news 
brief. These questions clearly refer to the motivations and 
biases of other experts. 

Example 

Would other professionals be biased to support this? 

o 19 
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1 .1 .4.6 Other Experts - Other 

This topic includes requests about the other experts in the 
research area described in the news brief but do not fit in the 
topics, 1.1.4.1 to 1.1.4.5. 



1.1.5 People - Other 

This topic includes specific and identifiable questions about people 
who may be associated to the research described in the news brief 
(e.g., have an opinion about the research described in the news brief) 
but are different from the people listed under topics 1.1.1 to 1.1.4 
(e.g., general public, consumers). 



1.2 SOURCE OF INFORMATION 

This section includes requests that pertain to the original or current 
publication outlet for the research described in the news brief (1.2.1) and 
the authors or broadcasters of the report (1.2.2), Questions categorized 
under 1.2.1 (Publication Outlet) may refer to the identity of the publication 
outlet, the credibility or prestige of the publication outlet, the biases of the 
publication outlet etc. Questions categorized under 1.2.2 
(Authors/Broadcasters) may refer to the identity of the authors or 
broadcasters, their credentials or training, their biases, etc. Specific and 
identifiable questions that refer to the source of information but are 
different from topics 1.2.1 or 1.2.2 are categorized under 1.2.3 (Source of 
Information - Other). 

AMBIGUOUS-Requests that refer to source of information but are too 
ambiguous to be categorized under topics 1.2.1, 1.2.2 or 1.2.3 (i.e. fit 
under more than one topic listed under Source of Information) are 
categorized as topic 1.2 (Source of Information). 



1.2.1 Publication Outlet 

This topic includes requests pertaining to the academic forum (ie. 
academic journal, conference presentation) or the nonacademic forum 
(i.e., news paper, TV broadcast) where the research in the news brief 
was originally or is currently presented. 

Examples 

Where was this research published? 

Was this research ever published in a peer-reviewed journal? 
What academic journal was this research presented in? 

9 20 
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Did this come from the National Inquirer? 
How prestigious is the journal? 



1.2.2 Authors/Broadcasters 

This topic concerns requests pertaining to authors of the original 
research report referred to in the news brief. Questions regarding the 
authors or broadcasters of the news brief that describes the research 
are also included under topic 1.2.2. 

Example 

Who wrote this report? 
Are the reporters biased? 

Are the reporters qualified to interpret this research? 



1.2.3 Source of Information - Other 

This topic concerns specific and identifiable questions pertaining the 
source of information but are different from topics 1.2.1. or 1.2.2. 



1.3 FUNDING ISSUES 

This topic concerns requests about the person(s) or institution that funded 
the research described in the news brief. Questions that focus on the 
researchers' or experimenters' motivations that may be influenced by the 
source of funding are not categorized under this topic but under topic 
1.1.1.5. (Researchers' Motivations, Expectations, Biases, Beliefs). 
Questions that focus on the funders' motivations, credentials, etc, are 
included under topic 1.3 Funding Issues. 

Example s 

Who paid for this research? 
Who was the granting agency? 

Why would anyone pay for this kind of research to be conducted? 
Who would be immoral enough to finance this type of research? 
Did the makers of Quipmanol fund this research? (P) 

1.4 IDENTIFICATION OF RESEARCH INSTITUTION 

This topic concerns requests about the institution where the research 
described in the news brief was based. These questions must specifically 
refer to the research institution (e.g., "At what university did this research 
take place?"). Questions about the particular setting that the subjects 
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were exposed to during the research are not included under this topic but 
under topic 3.2.2 (Setting). Also, questions about the general research 
context (i.e., lab versus field) are not included under this topic but are 
included under topic 3.1.2 (Research Context). Finally, ambiguous 
questions about the location of research (e.g., "Where was the research 
conducted?") are not included under this topic but under topic 8.0 
(Ambiguous but Relevant to the Research) because this question may refer 
to the research institution, the research context (lab versus natural 
environment) or to the setting, and not enough information is given to 
discriminate. 

Examples 

At what university are these researchers employed? 
At what university was this research conducted? 

Was this research conducted at the Alberta Environmental Centre or at a 
university? 



1.5 SOURCE OF RESEARCH QUESTION-IDENTIFICATION 

This topic concerns requests about the inspiration for the research 
described in the news brief. 

Examples 

What caused them to conduct this research in the first place? 

Where did they get the idea to study the effect of crystals on dreaming? 

Who started or thought of this? (P) 



1.6 SOCIAL CONTEXT - OTHER 

This topic includes requests that are related to the social context of the 
research described in the news brief but that are different from topics 1.1 
(People), 1.2 (Source of Information), 1.3 (Funding Issues), 1.4 
(Identification of Research Institution), or 1.5 (Source of the Research 
Question). 
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2.0 AGENT 

The agent is the treatment described in the news brief (i.e., the agent is the 
'thing' that produced the outcome). Exposing experimental subjects to an 
agent may involve exposing them to an object (e.g., crystal), a chemical (e.g., 
Permaldrin), or a process (e.g., meditation). Any changes that experimental 
subjects undergo are presumably caused by the agent. Included under this 
heading are questions that refer to the identification of the agent and its 
properties (2.1). Also included under this heading are questions that refer to 
the mechanism(s) by which the agent causes its effects and that refer to the 
nature of the effects/side effects (2.2). Finally questions that refer to 
alternative agents (2.3) are included under this heading. Specific and 
identifiable questions about the agent that are different from topics 2.1, 2.2 or 
2.3 are categorized under topic 2.4 (Agent - Other). 

Requests that refer to the agent described in the news brief (or similar 
agents) but that are too rudimentary or ambiguous to be categorized under 
topic 2. 1 to 2.4 (i.e., fit under more than one topic listed under Agent) are 
categorized as topic 2.0 (Agent). 



2.1 AGENT IDENTIFICATION/PROPERTIES 

This topic includes requests that are designed to get information about 
the agent independent of its mechanism, effects or side effects . The 
questions may be general in that they regard the general identification of 
the agent (e.g., "What is Permaldrin?"). Questions under this topic may 
also be specific in that they may regard the properties of the agent. 
Knowing about some of these properties may help one understand the 
mechanism of the agent (e.g., "What is Permaldrin composed of?"). 
Questions that refer to properties of the agent that are not relevant to its 
mechanisms, but may influence its utility (e.g., cost of agent, access, 
availability, safety or danger, functional use) are included under topic 5.2 
(Utility of Agent Application). Questions that refer to *he mechanisms, 
effects or side effects of the agent are not included un\.-.v this topic but are 
included under topic 2.2 (Agent Mechanisms, Effects/Side Effects). 
Typically these questions can be rephrased as "What is it?" questions. 

Examples 

What is Permaldrin? 

Where do you find Ollinite? 

What chemical elements make up Quipmanol? 

What's involved in Mai Handu Meditation? 

What are the steps involved in Mai Handu Meditation? 
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Where or what is this chemical found in? (P). 

2.2 AGENT MECHANISMS, EFFECTS AND SIDE EFFECTS (THEORY) 

This topic concerns requests about what the agent does, such as: (a)how 
the agent causes its effects or side effects and (b) what the nature of the 
effects or side effects are. Questions about how the agent causes its 
effects and side effects concern the theory of the mechanism(s) by which 
the agent produces the effect on the dependent variable. Such questions 
may refer to mediating variables that are necessary for the relationship 
between the agent and the outcome to hold. Questions about the effects 
and side effects concern the physical or behavioral outcome(s) that result 
from exposure to the agent. The "effects" refer to the dependent variable 
that the researchers were investigating. "Side effects" refer to aspects of 
the subject, other than the dependent variable, that are affected by 
exposure to the agent. Questions that address what the agent was 
designed to do (i.e., the goal that the creator had in mind when creating 
the agent, for example, the function of Permaldrin is to kill insects) are not 
included under this topic but are included under topic 5.2 (Utility of Agent 
Application). Questions that suggest that the negative side effects 
would detract from the utility of the agent are also coded as topic 5.2. 

Agent Mechanisms/Theory : These questions may refer to the actual 
mechanisms that underlie the manner in which the agent causes its 
effects or side effects or they may refer to the underlying theory which 
links the agent to the effects/side effects. Often these requests address 
"how" the agent does something. The subject may request or propose a 
logical connection between the agent and the outcome without introducing 
a confound. 

Examples 

How does Ollinite cause people to dream accurately about the future? 
Doesn't it seem counterintuitive that a poison reduces pollution? (This 
person is questioning the theory behind the claim.) 

Assuming that the conclusion is true, what is the causal link between the 

chemical and air pollution reduction (P) 

Do the 2 chemicals react with each other? (P) 

How can dreams predict the future? (P) 

How does Permaldrin enter the body of the robin? 

Effects/Side Effects : The effects are the dependent variables that 

were investigated by researchers before and after exposure to the agent. 

The side effects are aspects of the subject, other than ihe dependent 
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variable, that may have been affected by exposure to the agent. If the 
subject is suggesting that the side effects would detract from the ut.litv of 
the agent (i.e., costs outweigh the benefits), then the code is 5.2, Utility 
of Agent Application. 

Examples 

What does Permaldrin do to the body of a robin? 

What are the side effects of using Quipmanol? 

Does the crystal also affect their intelligence? 

Are there negative sids effects of using Quipmanol? (This mentions 

negative side effect which may be a 5.2 code but we are not sure Vvhy 

they mention the negstive side effects, so it is coded as 2.2) 

Does Quipmanol change other things in the car, like Antifreeze? (P) 

If you dissected the robin would you find anything unusual because of 

the insecticide? (P). 

Does it produce new pollution? (Note: There is no explicit reference to 
practicality or utility of agent application.) 



2.3 ALTERNATIVE AGENTS 

This topic includes requests that refer to other agents that may be similar 
to the agent described in the news brief. This topic may include general 
identification questions (e.g., "What kinds of crystals are there besides 
Ollinite that may have this effect?") or it may include specific questions 
about the properties, mechanisms or the effects and side effects of the 
alternative agents. The alternative agents were not involved in the 
research described in the news brief, thus they are not to be confused 
with confounding factors (see topic 3.1.4 Control of Other 
Factors/Confounding Factors). Questions that refer to alternative agents 
request information about the range of alternative forms of the agent that 
was described in the news brief. 

Examples 

Do other crystals have this effect on dreaming? 
What other pesticides are available? 
Are there other similar crystals? 

How is Mai Handu meditation different from transcendental meditation? 
(P) 

Was Mai Handu meditation the only type of meditation practiced? (P) 
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2.4 AGENT - OTHER 

This topic includes specific and identifiable questions about the agent 
described in the news brief (or similar alternative agents) but are different 
from topics 2.1 to 2.3. 
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3.0 METHODS 

Specific methodological questions may refer to the/esearch design (3.1), 
agent delivery (3.2), the subjects who participated in the study (3.3), the 
measures used in the study (3.4), consistency of methods (3.5) and/or 
replicability of methods or procedures (3.6). Specific and identifiable 
methodological questions that are different from topics 3.1 to 3.6 are 
categorized under topic 3.7 (Methods - Other). Questions that concern similar 
research not described in the news brief are not categorized under this heading 
but are included under Topic 6.0 (Related Research) or subtopics of topic 6.0. 

Requests that refer to methods (procedures, experiments) but are too 
ambiguous or rudimentary to be specifically categorized under topics 3.1 to 
3.5 (i.e., fit under more than one topic listed under Methods) are categorized 
under topic 3.0 (Methods). 

Examples 

What type of methods did they use? 
Did they use the scientific method? 
What type of procedure did they use? 
What type of experiment did they use? 
What was the nature of the research? (P) 

What experimental tools did the use to arrive at this conclusion? (P) 
How many times was the procedure repeated? (P) [Could be 3.1 
design or 3.6 replicability.] 



3.1 DESIGN 

This topic concerns requests about the design used to conduct the 
research described in the news brief. Questions regarding design entail 
general or specific questions about the type of design used (3.1 .1) and the 
research context (3.1.2). Also included under this topic are questions 
regarding the duration of the study (3.1 .3). Finally, questions may refer to 
the control of other factors (3.1 .4). Specific and identifiable questions that 
refer to design but are different from topics 3.1.1 to 3.1.4 are categorized 
as Design - Other (3.1.5). 

Requests that refer to design but are too ambiguous or rudimentary to be 
categorized under topics 3. 1. 7 to 3. 1.5 (i.e., fit under mure than one topic 
listed under Design) are categorized as topic 3. 1 (Design). 



3.1 .1 Tvne of Design 
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This topic includes requests intended to get information about the 
nature of the design used in the research described in the news brief. 
The questions may be genera! or specific. Note that ambiguous 
questions about methods (e.g., "What type of methods/experiments/ 
procedures did they use?") are not included under topic 3.1.1 (Type 
of Design) but are included under the superordinate heading 3.0 
(Methods). 

Examples 

What type of design did they use? 

Was this a correlational (observational) or an experimental study? 

Was this a cross-sectional or longitudinal study? 

Was this a repeated measures study? 

Is this a causation or correlational conclusion? (P) 

Were (a) the same robins observed before exposure and after or (b) 

was one group exposed vs another group-not exposed? (P) 



3.1.2 Research Context 

This topic includes requests about the general context of the 
research described in the news brief (e.g., laboratory context versus 
field context). Questions about the immediate setting to which the 
subjects were exposed (e.g., temperature, noise level) are not 
included under this topic but are included under topic 3.2.2 (Setting). 
Also, questions that refer to the research institution where 
research is based are not included under this topic but are included 
under topic 1.4 (Identification of Research Location). 

Example 

Was this research conducted in a laboratory or out in the field? 



3.1.3 Duration of the Research 

This topic includes requests about the period of time over which the 
research observations were made. However, questions about the 
duration of the effect are not included under this topic but are 
included under topic 4.1 .3 (Duration of Effect). Also, questions that 
refer to whether a longitudinal design was used are not included 
under this topic but are included under topic 3.1.1 (Type of Design). 

Examples 
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How long did they do this study? 

How long did they collect the measures for? (P) 

3.1.4 Control of Other Factors/Confounding Factors 

This topic concerns requests about other variables that should have 
been taken into account when designing the experiment such that 
alternative causes (i.e., not the agent) for the outcome can be ruled 
out. These questions must identify some connection to how the 
"other factor" could cause the outcome rather Jhan the agent. The 
questions may be general or they may refer to specific types of 
confounding variables (see examples below). The variables that are 
included under this topic are those that can be controlled for 
experimentally (e.g., use of placebos to control for subject 
expectation or motivation, consistent setting for all subjects). 
Confounding variables that can be statistically controlled are also 
included under this topic. Variables that cannot be controlled for 
experimentally (but may influence the results) are not included under 
this topic (e.g., see topic 1.1.1.5 Researchers' Motivations and/or 
Biases). Questions about mediating variables that are necessary for 
the relationship between the agent and outcome are not included 
under this topic but are included under 2.2 (Agent Mechanisms/ 
Effects/Side Effects). 

Examples: General Questions 

Could other factors have caused this? 

Was a control group used? 

Did they statistically control other variables? 

If the robins were exposed to any other external force, could the 

same reaction happen? (P) 

Examples: Specific Questions 

Could a placebo effect explain this effect? 

What about a placebo effect? 

Could it be that the people who wore the crystals produced more 
accurate dreams because of the instructions that they received? 
(procedure) 

Could it be that they achieved these results because the measuring 
instruments were unreliable? 

Could it be that the robins' age caused a decline in mating? (P) 
Could it be that pollution levels were decreasing naturally anyway? (P) 
Maybe they were mating less because there were more predators in 
the area? (P) 

Perhaps their science scores were higher becauso they tested the 
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students at the end of the school year? (calendar time) (P) 

Do you think the spread of environmentalism may be why the air is 

cleaner? (P) 

Could some other source cause the decrease in mating behavior? 



3.1.5 Design - Other 

This topic includes requests about design but that do not fit under 
topics 3.1.1 to 3.1.4. 



3.2 AGENT DELIVERY (PROCEDURE) 

This topic includes requests about manner in which agent was delivered 
to the subjects (3.2.1, e.g., instructions, dosage), the setting or immediate 
surroundings of the subjects (3.2.2), the time of year in which the research 
was conducted (3.2.3) and the consistency of agent delivery (3.2.4). 
Specific and identifiable questions that refer to agent delivery but are 
different from topics 3.2.1 to 3.2.4 are categorized as Agent Delivery- 
Other (3.2.5). 

Requests that refer to procedures but are too ambiguous or rudimentary 
to be categorized under topics 3.2. 1 to 3.2.5 (i.e., fit under more than one 
topic listed under Agent Delivery) are categorized as topic 3.2 (Agent 
Delivery). 

3.2.1 Nature of Agent Delivery 

This topic includes requests about the manner in which the agent 
was delivered to the subjects. 

Examples 

How frequently were they exposed to this pesticide? 
What kind of instructions did they give the subjects? 
What dosage did they use? 
What is the extent (area) of exposure? 

What is the duration of exposure? [Note: This question is different 
from "How long did they research this?" (3.1.3 Duration of 
Research).] 

How much Quipmanol is necessary to reduce pollution? (P) 
What was the level of exposure? (P) 



3.2.2 Setting 
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This topic includes requests that identify the immediate surroundings 
that suhjects were exposed to while being investigated. However, 
questions regarding the manner in which setting confounds the 
interpretation of the results are not included under this topic but 
under topic 3.1.4 (Control of Other Factors/Confounding Factors). 

Examples 

In what city did they investigate the effects of Quipmanol on air 
pollution levels? 

At what noise level were the subjects tested? 
At what temperature were the subjects tested? 
How many predators are there in the area? (P) 



3.2.3 Calendar Time 

This topic includes requests about when the research was 
conducted. These questions may be general (e.g., "When was this 
research conducted?") or they may be more specific (e.g., "Did they 
test students at the beginning or the year or at the end of the year?"). 
Questions that explicitly refer to how these time factors may affect 
the results (e.g., "Could it be that they studied the robins at a time 
when their reproductive rates are normally low anyway?") are not 
included under this topic but are included under topic 3.1 .4 (Control 
of Other Factors/Confounding Factors). 

Examples 

When was the research conducted? 

At what time of the year did they do this study? 



3.2.4 Agent Delivery - Other 

This topic includes specific and identifiable questions about procedure 
that are different from topics 3.2.1 to 3.2.4. 



3.3 SUBJECTS 

This topic includes requests about the subjects who participated in the 
research described in the news brief, Included under this topic are 
questions that refer to the identity of the subjects (3.3.1), the qualitative 
characteristics of the subjects (3.3.2), the number of subjects tested 
(3.3.3), the manner in which subjects were selected (3.3.4), the 
representativeness of the sample (3.3.5) and the expectations or 
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motivations of the subjects (3.3.6). Specific and identifiable questions that 
refer to subjects but are different from topics 3.3.1 to 3.3.6 are 
categorized under topic 3.3.7 (Subjects - Other). 

Requests that refer to subjects but are too rudimentary or ambiguous to 
be categorized under topics 3.3. 1 to 3.3. 7 (i.e., fit under more than one 
topic listed under Subjects) are categorized as topic 3.3 (Subjects). 



3.3.1 Identification 

This topic includes general questions about the identification of the 
participants (subjects). This topic does not concern specific 
questions about the characteristics of the participants (see topic 
3.3.2). This topic does not concern questions about researchers or 
participants other than subjects (see 1.0, Social Context). 

Examples 

Who were the subjects? 
Who did they study? (P) 



3.3.2 Qualitative Characteristics 

This topic includes specific questions about the qualitative 
characteristics of the participants (including age). However, 
questions that explicitly refer to how subject characteristics may 
affect the results are not included under this topic but under topic 
3.1.4 (Control of Other Factors/Confounding Factors), for example, 
"Maybe the students wno used the textbook did well on the 
Provincial exam because they were smarter to begin with." Also, 
questions that refer to the type of subjects for which the results are 
limited to are not included under this topic but are included under 
topic 5.1 (Generalizability of Agent Effects), for example, "How old 
you have to be to see this effect?" Finally, questions that refer to 
base rates or the levels of the dependent variable investigated in the 
study (e.g., "What were the mating behaviors like before exposure to 
Permaldrin?" or "How well did the students perform on the Provincial 
exam before reading this textbook?") are not included under this topic 
but are included under topic 4.1.1 (Absolute Nature of the Data). 

Examples 



How old were the subjects? 
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Where did the subjects come from? 

What breed of robin did they use? 

What kind of cars did they use in the study? 

Were the subjects happy people? 

Are the seniors also on medication? (P) 

What was the religious/social background of the people tested? (P) 



3.3.3 Number of Subjects Tested 

This topic includes requests about the number of subjects who 
participated in the research described in the news brief. 

Examples 

How many robins did they investigate? 
How large was their sample size? 



3.3.4 Sample Selection 

This topic includes requests that refer to how the sample used in the 
research described in the news brief was selected from the genera! 
population. These questions must specifically refer to the issue of 
selection. However, questions that explicitly refer to how sample 
selection may confound the results {e.g., "Did the researchers use 
volunteers who believed in the powers of crystals are thus were likely 
to give false reports of their dreams?") are not included under this 
topic but are included under topic 3.1.4 (Control of Other 
Factors/Confounding Factors). 

Examples 

How did they get these people? 

Did the people volunteer to participate in the study? 

Where did they get the subjects from? 

Did they randomly select their sample? 

Did they include younger subjects in their sample? 

Did the subjects volunteer to participate (i.e., self selection)? 



3.3.5 Representativeness of Sample 

This topic includes requests about whether the particular sample 
used in the research is representative of the population to which the 
results are to be generalized. These questions must explicitly refer to 
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the representativeness issue. However, questions that refer to the 
characteristics of the sample (e.g., "Did they test students from all 
economic levels?") but do not refer to the representativeness of the 
sample are not included under topic 3.3.4 but are included under topic 
3.3.2 (Qualitative Characteristics). 

Examples 

Is this sample representative of the population to which the results 
will generalize? 

Did they test students from all economic levels "to ensure that these 
result i/interpretation can be generalized to all students?" 



3.3.6 Expectations/Motivations/Awareness/Beliefs About the Design 

This topic includes requests about any expectations or motivations, 
that the participants may have in regard to the research described in 
the news brief. However, questions that explicitly refer to how 
expectations or motivations may bias the results (e.g., "Could it be 
that people who were wearing the crystal expected the crystal to 
work and interpreted their dreams as being accurate?") are not 
included under this topic but under topic 3.1.4 (Control of Other 
Factors/Confounding Factors). 

Examples 

What did the seniors expect to get out of this meditation? 

Did the subjects believe in the powers of the crystal? 

Were the people aware of why they were wearing the crystal? (P) 



3.3.7 Subjects - Other 

This topic encompasses specific and identifiable questions about the 
subjects but are different from topics 3.3.1 to 3.3.6. 

3.4 MEASURES 

This topic includes requests about the manner in which variables (i.e., 
dependent and independent) were measured in the research described in 
the news brief. Included under this topic are questions that refer to the 
identification of measures (3.4.1), manipulation check (3.4.2), evaluation 
of measures (3.4.3), and use of additional measures (3.4.4). Specific and 
identifiable question that refer to measures but are different from topics 
3.4.1 to 3.4.4 are categorized as Measures - Other (3.4.5). 
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Requests that refer to measures but are too rudimentary or ambiguous to 
be categorized under topics 3.4. 1 to 3.4.5 (i.e., fit under more than one 
topic listed under Measures are categorized as topic 3.4 (Measures). 



3.4.1 Identification of Measures 

This topic includes requests about the identification of the measures 
used in the research described in the news brief. Questions about the 
identity of the measures used in the research may be basic 
identification questions or they may be more technical. Technical 
questions may pertain to the operational definition or refer to how or 
whether the levels of dependent variable were defined in observable 
terms. Questions not included within topic 3.4.1 refer to the quality 
of the measures (see topic 3.4.3 Evaluation of the Measures) or refer 
to the data collected from the measures (see topics under the heading 
4.0 Data/Statistics). 

Examples: Basic Identification 

What were the measures? 

Did they use reaction times or error rates as their dependent 
measure? 

Did they use self-reports? 

How were the dreams interpreted? (P) 

What was the measure or criterion used to determine if meditation is 
beneficial? (P) 

How did they measure pollution levels? (P) 

Examples: Technical-Operational Definition of Dependent Variable 

How is sense of well-being operationally defined? 
What is your definition of well-being? (P) 



3.4.2 Manipulation Check 

This topic includes questions about how or whether the independent 
variable was measured in the research described in the news brief. 

Example 

Was the level of Quipmanol measured for each treatment condition? 



3.4.3 Evaluation of the Measures 
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This topic includes the quality of the measures used in the research. 
^These questions refer to the reliability, validity and precision of the 
measurement instruments. Questions that explicitly refer to how the 
lack of measurement reliability, validity and precision confounds the 
results (e.g., "Could it be that the researchers found this difference 
only because they couldn't measure 'sense of well-being' very 
accurately?") are not included under this topic but under topic 3.1.4 
(Control of Other Factors/Confounding Factors). 

Examples 

Were they really measuring sense of well-being? 
Were the measures valid? 

Was the instrument that measured pollution reliable? 

How accurate are the dreams? 

Did the dreams come true? 

How broad/specific were the predictions? 

How does one know whether the dreams predict the future? (P) 

Are the dreams general or specific? (P) 



3.4.4 Additional Measures 

This topic includes requests about how or whether the dependent 
variable described in the news brief was measured in other ways 
(e.g., "Did they also use the 'Well-Life Expectancy' scale along with 
the self reports?"). Questions under this topic refer to measures of 
the dependent variable only. However, questions about the measures 
of how the agent affects other aspects of the subjects used in the 
research (i.e. side effects) are included under topic 2.2 (Agent 
Mechanisms, Effects/Side Effects), for example, "Did they look at how 
their physical health is affected by meditation?" Also, questions that 
refer to the data collected from these other measures (e.g., "What 
were their scores on the ' We 1 1 Life Expectancy' measure?") are not 
included under this topic but are categorized under topic 4.1.4 
(Additional Data). 

Examples ^ 

Did they get a psychologist's assessment of the seniors' sense of 
well-being? 

How far in to the future were the predictions for? (P) 
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3.4.5 Measure^ - Other 

This topic concerns specific and identifiable questions about the 
measures used in the research described in the news brief, but are 
different from topics 3.4.1 to 3.4.5. 

Example 

How vigilant were the observations? (P) 

3.5 CONSISTENCY OF METHODS/PROCEDURES 

This topic refers to the consistent use of methods or procedures for all 
subjects who were involved in the research described in the news brief. 
Questions that refer to consistent methodology may be general or specific in 
nature. Not included within topic 3.5 are questions that refer to how an 
inconsistent methodology may confound the results but are included under 
topic 3.1.4 (Control of Other Factors/Confounding Factors). 

Example: General Methodological Consistency 

Did they use consistent methods/procedures for all subjects? 

Examples: Specific Methodological Consistency 

Did all subjects get the same dose? (Agent Delivery) 

Were all the subjects exposed to the same setting? (Setting) 

Did they use the same Provincial exam for all students? (Measures) 



3.6 REPLICABILITY OF METHODS/PROCEDURES 

These questions refer to whether the methodology can be replicated by 
other researchers or experimenters. Questions within this topic generally 
address issues about whether the experiment can be replicated. However, 
questions that refer to data are not included under this topic but are 
included under topic 6.3 (Supporting Data/Data Replication). 

Example 

Can other researchers replicate the experiment? 



3.7 METHODS - OTHER 

This topic refers to specific and identifiable questions that are about 
methodology but are different from topics 3.1 to 3.6. 
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4.0 DATA/STATISTICS 

This topic includes requests about the data collected in the research 
described in the news brief (4.1) and the statistics used to analyze the data 
(4.2). Specific and identifiable questions that refer to the data or statistics but 
are different from topics 4.1 and 4.2 are categorized as topic 4.1.5 (Data - 
Other) or topic 4.2.3 (Statistics - Other). 

Requests that rvfer to data or statistics but are too rudimentary or ambiguous 
to be categorized under topics 4. 1 or 4.2 (i.e., fit into more than one topic 
listed under Data/Statistics) are categorized as topic 4.0 (Data/Statistics). 

Example 

What proof do you have to back this up? 



4.1 DATA 

Specific questions about the data may refer to the absolute nature of the 
data gathered for the dependent variables (topic 4.1.1), the comparative 
nature of the data gathered for the dependent variable (topic 4.1 .2), the 
duration of the effect (topic 4.1.3) and additional data that may have been 
gathered from additional measures, had they been used (topic 4.1.4). 
Specific questions about the data that do not fit into topics 4.1 .1 to 4.1 .4 
are categorized under topic 4.1.5 (Data - Other). 

Requests that refer to the data but are too ambiguous or rudimentary to 
fit under topics 4. 1. 1 to 4. 1.5 (e.g., "What do the data look like?") are 
categorized under topic 4. 1 (Data). 

E xamples 

What do the data look like? 
What are the data? 



4.1.1 Absolute Nature of the Data for the Dependent Variables 

This topic includes requests about the absolute nature of the data 
collected in regard to the dependent variable defined in the research. 
These questions may refer to quantitative data (e.g., "What were the 
test scores on the provincial exam?") or qualitative data (e.g., "What 
was the content of their dreams?"). Questions about base rates 
regarding the dependent variable are also included under this topic 
(e.g., "Did they dream about the future before they wore the 
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crystal?" or "How often did they dream about the future before they 
wore the crystal?"). However, questions that refer to data about the 
effects of the agent on variables other than the dependent variable 
(e.g., "How did they score in math after using the Science for Life 
textbook?") are not included under this topic but are included under 
topic 2.2 (Agent Mechanisms, Effects/Side Effects). Also, questions 
that refer to the comparative nature of data are not included under 
this topic but are included under topic 4.1 .2. Questions about 
comparative data concern how the scores for the dependent variable 
changed before and after treatment or how the scores of the different 
treatment groups differed. Also, questions about data that may have 
been collected from additional measures, had they been used, are not 
included under this topic but are included under topic 4.1.4 
(Additional Data). 

Examples 

What were the scores on the Provincial exam after reading the text? 

What was the base reproductive rate of the robins? 

What was the content of their dreams? 

How frequently did they dream about the future? 

Out of the individuals examined, how many benefited and how many did 
not? (P) [Note that the request refers to absolute data i.e., frequency 
data, not comparative data.] 

What was their sense of well being before meditation (base rate)? (P) 
What did they predict? 

Can I see some examples of the dream reports? 

4.1.2 Comparative Nature of the Data for the Dependent Variable 

This topic includes requests that refer to between-subject, between- 
group, or within-subject comparisons in reference to the effect of the 
agent on the dependent variable. This topic includes requests about 
the size of effect (e.g., statistical significance of the results). These 
questions must explicitly refer to the comparative nature of the data. 
For example, the question "By how much does the current 
reproductive rate differ from the base rate?" would be categorized 
under topic 4.1.2. (Comparative Nature of Data) whereas the question 
"What was the base reproductive rate before exposure to 
Permaldrin?" would be categorized under topic 4.1.1 (Absolute Nature 
of the Data). Questions regarding statistical significance are included 
under this topic. Typically these requests ask for the differences in 
the effect before and after the agent was administered or 
comparisons of the effects of different agents. 
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Examples 

Were the base reproductive rates of the robins significantly higher 

before they were exposed to Permaldrin? 

Were these results statistically significant? 

How much more higher is the seniors' sense of well being after 

meditation as compared to before the meditation? 

How large of a reduction in air pollution? (P) 



4.1.3 Duration of Effect 

This topic includes requests about the duration of the effect of the 
treatment on the dependent variable. It does not include questions 
about the duration of the study (see topic 3.1.3 Duration of the 
Research) or duration of exposure to the agent (see topic 3.2.1 
Nature of Agent Delivery). 

Example 

For how long were the pollution levels suppressed by Quipmanol? 



4.1.4 Additional data 

This topic includes requests about the data that would have been 
collected about the dependent variable had additional measures been 
used in the research described in the news brief. Also included here 
are requests for information about the dependent variable that may be 
collected without "forma! measures" (e.g., "How long before you see 
an effect?"). However, questions that refer to additional measures 
(e.g., "Did they also use the 'Well-Life Expectancy Index' to 
measure sense of well-being?") are not included under this topic but 
are included under topic 3.4.4 (Additional Measures). Also, requests 
for information about side effects (i.e., not the dependent variable) 
are not included here but are included under topic 2.2 (Agent 
Mechanisms, Effects/Side Effects). 

Exampies 

What were the seniors' scores on the Well-Life Expectancy Index? 
How long before you see an effect? (P) 



4.1.5 Data - Other 

This topic includes specific and identifiable questions about the data 



40 



Scientific Literacy 
39 

available in the experiment but are different from topics 4.1.1 to 
4.1.4. 



4.2 STATISTICS 

Specific questions about the statistics used to analyze the data may refer 
to the nature of the statistics used in the research described in the news 
brief (topic 4.2.1) or to the appropriateness of the statistics (topic 4.2.2). 
Specific and identifiable questions that refer to statistics but are different 
from topics 4.2.1 or 4.2.2 are categorized under topic 4.2.3 (Statistics- 
Other). 

Requests that refer to the statistics but are too ambiguous or rudimentary 
to fit under topics 4.2. 1 to 4.2.3 (i.e., fit under more than one topic 
listed under Statistics) are categorized under topic 4.2 (Statistics). 

Example 

What about the statistics? 



4.2.1 Type/Nature of Statistical Analysis 

This topic include': requests about the type of statistical analysis that 
was performed in the research described in the news brief. 

Examples 

What kind of analysis did they perform? 

How powerful was the statistical test? 

Is the statistical test sensitive to interaction effects? 



4.2.2 Appropriateness of Statistics 

This topic includes requests about the appropriateness of the 
statistical analysis used on the data collected in the research 
described in the news brief. 

Examples 

Did they do a t-test when they should have used an ANOVA? 

Did they assume this causal relationship from a correlational analysis? 

4.2.3 Statistics - Other 



41 



Scientific Literacy 



40 



This topic includes specific and identifiable questions about the 
statistical analysis that are different from topics 4.2.1 to 4.2.2. 
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5.0 RELEVANCE OF THE AGENT/RESEARCH ON THE AGENT 

This topic includes requests about the relevance of agent above and beyond 
the research findings described in the news brief (i.e., beyond the effects of 
the agent on the particular subjects that participated in the research described 
in the news brief). Included under this heading are questions that refer to the 
generalizability of the agent effects to other subjects, species, or environments 
(5.1), the utility or practicality of using the agent (5.2), the recency of the 
research (5.3), the importance of the research to an audience of consumers or 
researchers (5.4), the contribution of the research to the domain of study 
(5.5), and audience familiarity with the research (5.6). Specific and 
identifiable questions that refer to the relevance of the research but are 
different from topics 5.1 to 5.6 are categc.ized under topic 5.7 (Relevance - 
Other). 

Requests that refer to the relevance of the research but that are too 
ambiguous or rudimentary to be categorized under topics 5. 1 to 5.7 (i.e., fit 
under more than one topic listed under Relevance of Research) are 
categorized under topic 5.0 (Relevance of the Agent/Research on the Agent). 

Examples 

Why hasn't this been put in all gasses? (P) [Note that this is 

an ambiguous request as it could be coded as either 5.4 or 5.6.] 

What will your company do to prevent accidents from occurring from this 

poisonous chemical? [Note that this is an ambiguous request as it could be 

coded as either 5.2 or 5.4.] 



5.1 GENERALIZABILITY/SPECIFICITY OF AGENT EFFECT 

This topic includes requests about whether the effects observed in the 
reported research would occur outside the particular research described in 
the news brief (i.e., generalize to other species and/or other environments). 
Also included under this topic are questions about the specificity of the 
effect to the subjects and environment used in the reported research. 
Typically these questions try to establish the boundary conditions (i.e., 
necessary conditions) under which the relationship between agent and 
outcome holds. 

Examples 

Are other birds affected by Permaldrin in the same way? 
Are only robins affected? 
Why are only robins affected? 
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Do you see this effect in humans as well? 

Does this effect occur at lower doses? 

Will you see this effect in the classroom setting? 

How old do you have to be in order to have Mai Handu meditation 

increase your sense of well-being? (P) [Note that the question "How 

old were the seniors?" would be coded as topic 3.3.2, Qualitative 

Characteristics.] 

Was the reaction the same for both males and females? (P) 
What kind of gasoline does it have to be burned with? (P) 
Does a certain kind of car have to be used to see this effect? 
Does the subject have to believe in the crystal for it to work? 



5.2 PRACTICALITY/UTILITY/FUNCTION OF AGENT APPLICATION 

This topic includes requests about the practicality of applying or using the 
agent in the real world (i.e., beyond the confines of the research 
environment). Included under this topic are questions about the costs or 
benefits of the agent, the accessibility or availability of the agent, the 
safety or danger of using the agent, and whether there is a need or 
demand for the agent. Questions may include requests about who will use 
the agent, what use the agent will be, or mention factors that would 
detract from the utility or application of the agent even if the effects were 
reliable. Questions about the importance of the finding to people are 
included within topic 5.4 (Impact of the Agent/Research Findings/ 
Conclusion). 

Examples 

Would it be practical to get all seniors to try this meditation? 

Do the costs of Quipmanol outweigh the benefits? 

Do we really need to use Permaldrin-is it worth it? 

How expensive is Quipmanol? 

Do we need Quipmanol to reduce pollution? 

How readily available is the chemical in terms of abundancy? (P) 

How much does it cost? (P) 

Why would we want to poison the air in order to reduce pollution? (P) 
[Note that this subject has accepted the theory behind the research 
but the practicality of the research is questioned here. If the question 
were "Doesn't poison pollute the air?", it would be categorized as topic 
2.2 (Agent Mechanisms, Effects/Side Effects) because the theory is 
brought into question.] 

Who would handle Quipmanol, the driver or the gas company? (P) 
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5.3 RECENCY OF RESEARCH 

This topic includes requests about how current the research described in 
the news brief is. However, questions that refer to time-related issues 
(e.g., "When was this research conducted?") but do not explicitly refer to 
how current the research is are not included under this topic but under 
topic 3.2.3 (Calendar Time). 

Examples 

Is this research up to date? 

Is their research approach up to date? 



5.4 IMPACT OF THE AGENT/RESEARCH FINDINGS/CONCLUSION 

This topic includes requests about the importance of the research findings 
to an audience of either researchers or general public. This topic includes 
questions about the audiences' attitude towards the research and how 
they have reacted to the research (i.e., changed their behavior as a 
consequence of the research). Subjects typically assume the research is 
true and try to establish what influence it will have based on people's 
reactions. However, questions that specifically address whether or not 
people will use the agent are coded topic 5.2 (Practicality/Utility/ Function 
of Agent Application). 

Examples 

Who cares about the research? 

Do other researchers recognize this research finding as being important? 
What kind of impact will this research have on our behaviors-that is will 
researchers try and find another way to deal with insects other than with 
Permaldrin? 

Have insecticide manufacturers responded to this research by not making 
Permaldrin? 

Why would anyone want to study the effects of crystals? 

Are gas manufacturers going to add Quipmanol to their gas as a result of 

this research? 

Does adopting this meditation mean sacrificing other beliefs? (P) 

Is the decline of robins a problem? (P) 

Why is this pesticide still being used? (P) 

What will be done to improve the situation of the robins? (P) 

How dangerous is the current level of air pollution? (P) 

Are there any immediate ways that we can reduce the chemical? (P) 
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Would robins still be successful in keeping their population at a steady 
level even though they are mating less? (P) 

5.5 CONTRIBUTION OF KNOWLEDGE THIS RESEARCH ADDS TO THE 
DOMAIN 

This topic includes requests about how or whether the research findings 
add to what researchers already know about the domain, how or whether 
the findings will affect the theories within the domain, or how or whether 
the findings will affect how other researchers will study the domain in the 
future. Questions in this area address whether the findings will change 
beliefs or existing knowledge in the area. 

Examples 

How does this research add to what we know about seniors' sense of 
well-being? 

Will these research findings modify any theories of seniors' sense of well- 
being? 

5.6 AUDIENCE FAMILIARITY WITH THE AGENT/RESEARCH FINDINGS 

This topic includes requests about how well-known the agent or research 
is to either researchers or to the general public. 

Examples 

Why haven't I heard of Quipmanol if its so important to controlling the air 
pollution? 

Is the popularity of this growing? (P) 

Do many people practice this type of meditation? [Note that this does not 
refer to the number of subjects who were exposed to the meditation in the 
research (topic 3.3.3, Number of Subjects) but the number of people 
outside the research context who practice the meditation. This form of 
participation is considered to reflect audience familiarity with the agent.] 



5.7 RELEVANCE - OTHER 

This topic includes specific and identifiable questions about the relevance 
of the research that are different from topics 5.1 to 5.6. 
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6.0 RELATED RESEARCH 
(BEYOND THE STUDY DESCRIBED IN THE NEWS BRIEF) 

This topic includes requests about other research that is related to the 
research described in the news brjefs. The other studies may be conducted by 
other researchers or by the same researchers described in the news brief but 
at a different time. Questions may specifically refer to the domain of study 
(6.1), the method of study (6.2), supporting data or data replication (6.3) or 
consensus (6.4). Specific and identifiable questions that refer to related 
research but are different from topics 6.1 to 6.4 are categorized under topic 
6.5 (Related Research - Other). 

Requests that refer to related research but are too ambiguous or rudimentary 
to be categorized under topics 6. 1 to 6.5 (i.e., refer to more than one topic 
listed under Related Research, for example, "What about other research?" are 
categorized as topic 6.0 (Related Research). 



6.1 SIMILAR DOMAIN OF STUDY 

This topic includes requests about whether other studies have been done 
in the same domain and/or the number of other studies that have been 
done in the domain. This topic does not include questions about whether 
the same methods were used (see topic 6.2, Method of Study), whether 
supportive data from these studies have been collected (see topic 6.3, 
Supporting Data/Data Replication), nor consensus among other researchers 
about the conclusion (see topic 6.4, Consensus/Nonconsensus). 

Examples 

Have other researchers studied the effects of meditation on seniors' sense 
of well being? 

How many studies have been done in this area? 

6.2 METHOD OF STUDY 

This topic includes requests about the methods used by other researchers 
who have studied the domain. Included under this topic are questions 
about the type of design, procedure, or measures that were used. 

Examples 

How did other researchers study the effects of meditation on seniors' 
sense of well-being? 

Have other researchers measured the effects of meditation on seniors 
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sense of well-being in the same way? 



6.3 SUPPORTING DATA/DATA REPLICATION 

This topic includes requests as to whether similar results have been found 
in other similar studies. These questions may refer to data replication or to 
other data that supports the conclusion reported in the news brief. The 
studies that produced the supporting data may have been conducted by 
the researcher described in the news brief or by other researchers. Also 
included under this topic are questions about whether there are data from 
other studies that contradict the data collected in the research described in 
the news brief. However, this topic does not include questions about the 
reliability of measures (see topic 3.4.4, Evaluation of Measures). Also, 
questions that refer to consensus (agreement with interpretation) are not 
included under this topic but are included under topic 6.4 (Consensus/ 
Nonconsensus). 

Examples 

Have these results been attained in other studies? 

Do other researchers get the same results? 

Do other researchers get different results? 

Have these results been replicated? 

Do data from other studies support these finding? 

Were the findings proven by other researchers in a controlled lab? 

Have other researchers achieved similar results in a lab experiment (P) 



6.4 CONSENSUS/NONCONSENSUS: AGREEMENT OR DISAGREEMENT WITH 
THE CONCLUSION 

This topic includes requests about whether other experts (i.e., other 
researchers, promoters or detractors, creators of the agent) agree with the 
interpretation of the data collected in the research described in the news 
brief. General questions that refer to other experts' opinions but do not 
specifically refer to their opinion about the research conclusion (e.g., 
"What do other researchers have to say about this?") are not included 
under this topic but are incluoed under topic 1.1.4.3 (Other Experts- 
Testimonials). 

Examples 

Do other researchers agree with this? 

Do other researchers agree that meditation is effective in increasing 
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seniors' sense of well being? 

Do other researchers agree that Permaldrin is the cause of decreased 
mating in robins? 

Do the researchers agree with the conclusion of Members for Autos for 
the Future? (P) 



6.5 RELATED RESEARCH - OTHER 

This topic includes specific identifiable questions about related research 
that are different from topics 6.1 to 6.4. 
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7.0 OTHER 

This topic includes specific identifiable questions that are relevant to the 
research described in the news brief but they are different from any of the 
topics outlined (1.0, 2.0, 3.0, 4.0, 5.0, and 6.0). These questions have 
not received a specific topic designation because they were asked 
infrequently. In future studies, specific and identifiable topics included 
under topic 7.0 (Other) may warrant a separate topic designation. (For 
now, include questions about disbelief here. If they are frequent in other 
studies, a separate topic can be created.) 

Examples 

Is the theory behind this research (crystals-dreaming) falsifiable? 
Can this be proven scientifically? 



8.0 AMBIGUOUS 

BUT RELEVANT TO THE RESEARCH DESCRIBED IN THE NEWS BRIEF 

This topic includes requests that are relevant to the research described in the 
news brief but are ambiguous because they fit under two or more major topic 
headings (1.0, 2.0, 3.0, 4.0, 5.0 or 6.0). For clarification on "ambiguous", see 
Principle 3. 

Examples 

How did they determine this? 

How did they come to this conclusion? 

How much research has gone into this? (Within study versus beyond study) 
Where was this research conducted? [Social Context (research institution) 
versus Methods (research context or setting)] 

When was this research conducted? [Methods (calendar time) versus 
Relevance of Research (currency of research)] 

What was the treatment? ("treatment" may refer to Agent or to Method) 
How long had meditation been practiced for? [Method (subject characteristics 
or duration of exposure) versus Agent (properties)] 

Why are they less likely to mate? (or) How is sense of well-being increased? 
[Note: These two questions do not refer to the agent so they cannot be 
categorized under 2.2. It is unclear what the subject was referring to when 
making the requests so they are coded as ambiguous.] 



9 

ERIC 



50 



Scientific Literacy 
49 

9.0 OFF-TASK (IRRELEVANT TO THE STUDY) 

This topic includes requests that are not related to the research described in 
the news brief. 

Examples 

Are robins' mating behaviors really decreased after being exposed to 
Permaldrin? 

Does Permaldrin really exists? 
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ASSIGNMENT OF KNOWLEDGE DOMAINS 

The goal of this section is to outline the procedure for assigning knowledge 
domain codes to the justification of each request. This procedure usually 
requires an analysis of the response that follows prompts like, "How would 
an answer to this "question help you to decide whether the underlined 
conclusion in the news brief is true?" Justifications should reflect the types 
of knowledge about scientific research that are the reasons or motivations 
behind the requests. Knowledge domain codes are intended to reflect the 
subjects' knowledge or understanding of why the information requested is 
important for evaluating scientific research. 

The basic procedure for identifying knowledge domains underlying the 
request is similar to the procedure for assigning a topic. Initially, select a 
general heading and then try to identify the most specific category available. 
However, some level of inference may be required to identify the appropriate 
knowledge domain and a request can arise from a variety of knowledge 
domains. As with coding topics, it is important not to add anything to the 
response that may change it. Thus, the knowledge domain may be clearly 
expressed, may require some level of inference, or may remain unknown. 
When the reason for making the request can not be identified the justification 
should be coded as 6.0 (No Identifiable Knowledge Domain). 

STEPS FOR ASSIGNING A KNOWLEDGE DOMAIN CODE 

REMEMBER PRINCIPLE 1: Assume the participant is doing the task. 

1) Read the justification that appears after prompts like: "How would an 
answer to this question help you to decide whether the underlined 
conclusion in the news brief is true?" 

REMEMBER PRINCIPLE 2: Know/edge domain codes should be made 
based on the information in both the justification and related request. 

NOTE: The use of a request may be essential to interpret the 
justification because it is conventional in discourse not to fully repeat an 
idea that has just been mentioned. Thus, interpreting the justification 
may require establishing the referential coherence between nouns in the 
request and pronouns in the justification. 

2) Decide how the justification relates to the request. In some way, the 
justification must refer to the request or provide some link to the request 
in order to be coded as a justification. To find this link it may be 
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necessary to paraphrase the response. Be especially careful when 
paraphrasing not to add anything to the response that was not part of 
the initial statement. 

3) To assign a knowledge domain code, decide which of the 'top level' 
categories encompasses the justification and related request. That is 
decide whether the justification is related to knowledge about "good" 
scientific practices and conventions (1.0), knowledge about the social 
context of science (2.0), knowledge about the function of science in 
society (3.0), is some other knowledge domain (4.0), is ambiguous but 
relevant (5.0) or is completely off-task and has no identifiable 
knowledge domain (6.0). 

REMEMBER PRINCIPLE 3: There are only three classes of justifications. 
Justifications may be: (a) rudimentary or ambiguous with respect to the 
categories listed; (b) representative of one of the categories listed; or (c) 
representative of an "other" category. 

REMEMBER PRINCIPLE 4: Know/edge domain codes can be assigned 
to justifications on the basis of a strict or lenient criterion. 



ERIC 



4) Once general area is identified, examine the subcategories within that 
section to determine whether or not the justification can be classified 
further. Use the most specific level of classification into which the 
justification clearly fits . 

a) Recall, from the description of Principle 3, that justifications that 
are too rudimentary or are too ambiguous to be given a specific 
category knowledge domain code should be given a knowledge 
domain code that reflects a more general level of classification. 
Note that if the justification is too rudimentary or ambiguous with 
respect to the top-level categories, the justification should be coded 
as 5.0 (Ambiguous but Relevant). 

b) If justifications reflect information that is not captured by the 
specified categories, code the justification as an "other" category in 
line with the top-level category that best describes it. Note that if 
there is no top-level category that reflects the gist of the justification 
then the justification should be coded as 4.0 (Other). 

c) If the justification clearly addresses more than one specific 
knowledge domain, it will be necessary to separate the justification 
into separate ideas according to the guidelines set out in the section 
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on "Identification of Requests and Justifications", and code each 
idea separately. 

NOTE: It may be difficult to decide whether the justification reflects 
two separate know/edge domains or is ambiguous. If you can code 
the justification as knowledge domain 'a' AND knowledge domain 
'b' then it reflects two ideas; if you can code the justification as 
knowledge domain 'a' QR knowledge domain 'b' then it is 
ambiguous. 

d) Sometimes instead of providing a justification the participant 
requests new information in response to the second prompt. In 
these instances the new request should be- given a topic code and 
the old request should be given a knowledge domain code 6.0 (No 
Identifiable Knowledge Domain). 

e) The participant may also justify the request in response to the 
first prompt and a response to the second prompt would be 
redundant. In these instances both the topic and knowledge domain 
code can be assigned based on the response to the first prompt. 

Example 

Did they take other factors into cons. Jeration such that they know 
that Permaldrin caused the effect and not other factors? 

NOTE: Examples provided with each knowledge domain description 
in the next section are not exhaustive and justifications may 
represent the same idea in different ways. 

NOTE: Every idea which receives a topic code must receive a 
knowledge code that represents the justification for the request. If 
no justification is provided a know/edge domain code of 6.0 (No 
Identifiable Knowledge Domain) should be assigned. A summary of 
knowledge domain codes is provided in Appendix 3. 
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KNOWLEDGE DOMAIN DESCRIPTIONS 

Whereas topic refers to the content of the question posed by participants, 
knowledge domain refers to the kinds of knowledge people use to evaluate 
reports of scientific research. Major categories include knowledge about 
"good" scientific practices and research (1.0), knowledge about the social 
context of science (2.0) f knowledge about the functions of science in 
society (3.0). Responses that clearly are relevant but that do not fit into 
any of these categories should be coded as other (4.0). Responses that are 
rudimentary or are ambiguous because they fit into more than one knowledge 
domain are coded as ambiguous (5.0). Some responses do not represent any 
knowledge domain (6.0). 

Knowledge domains 1.0 to 3.0 are organized hierarchically, from more 
general, superordinate categories to levels of more specific subordinate 
categories. Comments should be coded using the most precise subcategory 
possible. For example, if a person makes a remark that clearly indicates 
knowledge about the importance of sample size for inferences from data, the 
appropriate coding would be 1.6.1 (see codes below). If, however, the 
remark clearly indicates knowledge about sample size and effect size, then 
two separate codes would be appropriate (i.e., 1.6.1 and 1.6.2). If the 
remark relates to the immediately superordinate category but is different than 
any of the defined subordinate categories, it should be coded as other 
(i.e., 1.6.5). Finally, a remark may be clearly related to the superordinate 
category but it may be too ambiguous (i.e., it relates only partially to one or 
more subordinate categories) or too rudimentary (i.e., it represents 
knowledge related to the superordinate but is not sufficiently sophisticated or 
distinct to be assigned a subordinate code) to be categorized more precisely. 
In these cases the appropriate code would be 1.6 (the superordinate 
category). 
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1.0 KNOWLEDGE ABOUT "GOOD" SCIENTIFIC PRACTICES AND 

CONVENTIONS 

Knowledge about "good" scientific practices and conventions includes what 
people know or believe about the proper conduct of research. Included is 
knowledge about the necessity of using an adequate research design (1.1), 
the role of theoretical explanation (1.2), the value of controlling for 
confounding variables in delineating the cause-effect relationship between the 
putative agent and the outcome (1.3), the value of identifying other similar 
forms of the agent that have identical effects (1.4), the processes of 
measurement (1.5) and statistical inference (1.6), the influence, of consensual 
processes in the evaluation of data (1.7) and the significance of establishing 
boundary conditions under which the conclusion is valid (1.8). Other specific 
knowledge domains that are different from 1.1 to 1.8 are coded as 1.9 
(Other). Responses that reflect a rudimentary or ambiguous knowledge of 
good scientific practices are coded as 1 .0 (Knowledge About "Good" Scientific 
Practices and Conventions). 

Ex amples 

What was their method? It is important to use scientific methods. 

Do they have adequate proof to back up this claim. You have to p.ovide 
evidence before I believe the conclusion. 



1.1 ADEQUACY OF RESEARCH DESIGN 

Good scientists will use research designs that are adequate given the 
questions they are asking. For example correlational studies are 
appropriate for exploratory research, but a correlational design cannot be 
used when trying to establish a causal relationship. Included in this 
category are inquiries about the type of research design used and 
whether this type of design can support the type of conclusion being 
drawn. 

Example 

Was this an observational study or experimental study? If this was a 
correlational study the researchers should not make causal statements. 

Was this a longitudinal study? I would have more faith in the conclusion 
if they collected data over an extended period of time. 
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1 .2 THEORETICAL EXPLANATION 

People sometimes seek an explanation about how the agent or treatment 
could have produced the outcome. We interpret these requests as 
reflecting a belief that "good science" goes beyond merely noting 
probabilistic relations between agent and outcome but aiso seeks to 
establish a plausible description of the variables or mechanisms that 
underlie that relation. Included in this category are inquiries about how the 
agent or treatment might cause a certain outcome, about other variables 
that might intervene between the agent and outcome, and about other 
variables that might be necessary for the agent to produce the outcome. 
The common characteristic is that the person accepts the proposal that the 
agent causes the outcome and is seeking an elaboration on the relation 
between agent and outcome. This category is to be distinguished from 
1.3, where people suggest that other factors may have caused the 
outcome rather than the putative agent. 

Examples 

Why does this insecticide affect the mating? By understanding why 
something is the way it is, I would be more likely, to believe what is told to 
me. I need to understand the reasoning behind the concept. (P) 

What does Mai Handu meditation involve? The answer would help me to 
understand how the meditation might effect [sic] well being. (P) 

Why does it (meditation) increase their well being? If they can show that 
meditation (satisfies) some mental or emotional need, then I can see it as 
being trut\ (P) 

Why does this chemical cause a decline in the robins' mating behavior? If 
they can show that the chemical physically alters the brain or its structures, 
then that would make the conclusion more believable. (P) 

How does the chemical reduce the mating behavior of the robin? This 
would help me to understand exactly what was happening to the bird due to 
the chemical. (P) 

How are the 'Science of Life' books different from the other books used for 
the same course? Maybe the layout of the book helps students to better 
understand and remember the material, which would have an effect ( + ) on 
the provincial exam. (P) 
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1.3 CONTROLLING FOR CONFOUNDING VARIABLES/ELIMINATION OF OTHER CAUSES 

Inquiries often are made about the possibility that something other than the 
putative treatment or agent might be responsible for the outcome. These 
inquiries reflect knowledge that inferences about the relations between an 
agent and an outcome can be wrong, and that other factors may be the 
cause of the outcome. Alternative causes may include characteristics of the 
subjects in the research, variables other than the putative agent associated 
with the treatment, and other extraneous factors. Inquiries about mediating 
variables that intervene between the agents referred to in the news brief 
and the outcome are not included under 1.3 but are instead coded as 1.2 
Comments that seem to represent a mix of these subordinate categories or 
are too general to be classified in the subordinate categories should be 
classified as 1.3. 

Examples 

Was a control group used? To rule out other causes. (P) 

Could there have been any other reasons for the miscarriages of the rats? 
Other reasons may disallow the data. (P) 

What other factors could be involved in their conclusion? Maybe the birds 
have been exposed to other things which may have caused this 
conclusion. (P) [Note: This comment is coded as 1.3 because 
"other factors" could refer to pre-existing differences among subjects due to 
past exposure (1.3.1) or to confounding treatment variables (1.3.2).] 



1.3.1 Subjects 

Included here are comments indicating that pre-existing differences 
among subjects could have produced the outcome, such as 
motivations, incentives, expectations, biases, age, and other persona! 
characteristics. 

Examples 

Who were the subjects in the study? If the people in the study were 
all interested in psychic phenomena I would less value the outcome 
and believe that they made dreams turn out the way they wanted to, 
but the stones had no involvement. (P) 

Were all types of students with varying intellectual ability and 
capacity utilizing this research or were the types of students who 
used the 'Science for Life' text all on the same level? A student's 
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intellectual ability and capacity has an affect on exam marks with or 
without the 'Science for Life' book, therefore it could be the types of 
students that did well and not the book used to teach the material 
that made them do better. (P) 

What kind of subjects were chosen to participate in the experiment? 

Some people burn calories more readily and everyone has a different 

metabolism. Perhaps this is the reason they burned more calories and 

it had nothing to do with the music. (P) [Note: This response 

is coded as 1.3.1 rather than 1.3 because a clear indication 

is given that the source of the observed effect lies in the subjects 

who participated in the study.] 

What age group was tested? Age plays (an) important role in body 
metabolism and how many calories you burn. (P) 
[Note: Again, this response is coded as 1.3.1 rather than 1.3 
because a clear indication is given that the source of the observed 
effect lies in the subjects who participated in the study.] 

Did the subjects know what the meditation was supposed to do. A 
placebo effect may be the reason for an increased sense of well- 
being. 

1.3.2 Treatment 

Included here are references to characteristics of the treatment 
conditions, other than the putative agent, that could have produced 
the outcome. This category is restricted to comments in which 
subjects explicitly refer to variables that could be manipulated in an 
experiment. 

Example 

How quiet were their surroundings? Maybe their sense of well being 
increased because they were studied in a quiet room. 



1.3.3 Alternative Explanations - Other 

This category is for comments about variables, other than subjects or 
treatment, that could have produced the outcome. (Comments that 
partially fit more than one subordinate category would be scored as 
1.3, as would comments that are too vague to fit clearly in one of the 
more specific subordinate categories.) 
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Example 

Were there predators in the area? Maybe the robins mated less 
because predators imposed on their mating. 



1 .4 IDENTIFICATION OF AGENTS OR TREATMENTS WITH IDENTICAL EFFECTS 

Sometimes participants mention the value of finding an agent or treatment, 
other than the one in the report, that produces the same outcome as 
described in the report. To be coded 1.4, such comments should reflect 
knowledge that finding alternative agents can be important for identifying 
underlying mechanisms associated with that outcome. If the participant 
emphasizes the importance of finding alternative agents for practical 
reasons (e.g., for reducing costs or risks associated with producing that 
outcome), then the appropriate code is 3.1 (Utility for Application). 
Comments of this type also need to be distinguished from those involving 
alternative causes which may act as a confounding variable in the 
research (1.3, Controlling for Confounding Variables/Elimination of Other 
Causes). Finally, references to mediating variables that intervene 
between the agent and outcome are coded as 1 .2 (Theoretical 
Explanation). 

Examples 

Can other elements be mixed with zinc to form gold? If other elements 
had the same effect on the zinc then it would allow me to believe that 
there is some common factor that transforms zinc into gold. (P) 

Are there alternative insecticides that do the same job, but are less 
harmful? If a safer insecticide is used and robins' mating behavior isn't 
affected, then one knows Permaldrin is dangerous--it is the cause of 
decreases in mating behavior. (P) [Note: This person appears 
initially to be concerned with application (3.1), but in the 
justification it is clear that he or she is concerned primarily with 
identifying the cause.] 



1.5 MEASUREMENT 

Questions about measurement properties reflect knowledge that 
measures used in scientific research are not always appropriate. Specific 
subcategories include reliability and validity of measures. Often, 
however, questions and comments seem to represent a mix of these 
subordinate categories or are too general to be classified in the 
subordinate categories, and so 1.5 would be the appropriate 
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classification. 



Example 



How is a sense of well being measured? If it is by the researchers' own 
perception I am less likely to believe the conclusion than if it was 
measured scientifically. (P) [Note: The term "scientifically" 
is ambiguous.] 



1.5.1 Reliability 

Comments about the reliability (i.e., the self-consistency) of the 
measures are included here. 



1.5.2 Validity 

Participants often ask questions about whether the measures are valid 
in the sense that they accurately reflect what they are supposed to 
measure. We interpret these questions as reflecting knowledge about 
the importance of validity in measurement. 

Comments about the importance of using other measures to collect 
converging or diverging evidence are included here as well. Converging 
evidence refers to data from other, independent measures indicating that 
the dependent variable is measuring the same thing as are other measures 
of the same construct. Diverging evidence refers to data showing that 
the dependent measure is not related to different constructs. Comments 
about the utility of measures other than those described in the report 
should be coded as 1.5.2 as well. 

Note that this category is reserved for comments regarding "measures" 
within the reported study. Comments about converging "evidence" 
from other studies should be coded differently (see 1.7, Consensual 
Processes, for replication and for "fit" with evidence from other 
studies). 

Examples 

How do you measure sense of well-being? Validity of the measure 
may be an issue--you could be measuring something else. (P) 

What did the subjects in the study have to say? If they were 
positive about the effects of the treatment, I would find the 
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conclusion more compelling. 

How accurate were the dreams? If they were vague or abstract, I would 
not consider them to provide valid predictions of the future. 

How is well being measured? Well being is a subjective phenomena and 
would be hard to measure. 



1.5.3 Measurement - Other 

Included here are references to other specific measurement problems 
(e.g., scaling problems, floor or ceiling effects). 

Example 

"How can you measure a sense of well being? This would help me to 
see if there is a reasonable scale for well being to see if different 
values could be compared." (P) 



1.6 STATISTICAL INFERENCES ABOUT THE DATA 

Included are comments reflecting knowledge, technical or intuitive, about 
characteristics of the data or data analyses that influence the validity of 
inferences made from the data. Comments that seem to represent a mix 
of these subordinate categories or are too general to be classified in the 
subordinate categories should be classified as 1.6. Questions about 
base rates are also included under 1.6. 

Examples 

How is this possible? If this answer was scientific and was supported 
with evidence, I would be more likely to believe it. (P) 

Was there really a difference? If the difference they report is not real, 
then the conclusion will be misleading. [Note: The participant 
may be referring to more than one of the subordinate categories, but we 
cannot be sure which ones.] 

Is this different than the base rate? If not, then I would not believe the 
conclusion. [Note: Clearly the participant knows about the 
importance of an appropriate comparison, but again we cannot determine 
which of the subordinate categories, if any, the participant has in mind.] 
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1.6.1 Sample Size 

Included are remarks indicating knowledge that generalizations from 
the data are more likely to be valid when the sample size is relatively 
large (the law of large numbers). 

Example 

How many people were in the study? If there were many people in 
the study I would be more inclined to believe the 
underlined information. 



1.6.2 Effect Size 

Included are remarks indicating knowledge that generalizations from 
the data are more likely to be valid when the size of the effect (i.e., 
the difference between treatment and control groups, or between pre- 
treatment and post-treatment data) is relatively large. 

Examples 

How much would it reduce the level of pollution? If it reduces 
pollution a lot then it would seem to be true, if it doesn't it would be 
false. (P) 

How many birds were tested/ This would help me by showing that 
in many cases this occurred or in few cases this occurred. Many --> 
true, few --> false. (P) 

How much of decline in mating behavior? If the decline is small, then I 
would find the statement false in the sense that it is 
oversensationalized. (P) 



1.6.3 Variability 

Included are comments indicating that variability in the data can have 
an important influence on conclusions about mean differences 
between groups. 

Example 

How variable is this characteristic in the general population of 
seniors they sampled from? If all the seniors in the area have a high 
sense of well being, then the meditation isn't making much of a 
difference. 
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1.6.4 Statistical Significance and Chance 

Included are remarks indicating knowledge that differences (e.g., 
between groups) could be due to chance or measurement error, or 
that a test of statistical significance is needed to support the 
conclusion. 



1.6.5 Statistical Inference - Other 

Included here are references to other specific issues concerning 
statistical inference that do not fit into categories 1 .6.1 to 1 .6.4. 



1.7 CONSENSUAL PROCESSES 

Included in this category are comments reflecting an understanding that 
judgments about validity of findings and conclusions depends to a large 
extent on consensual agreement among researchers. In the scientific 
community consensual agreement depends in large part on whether a study 
has been replicated and on the degree to which the results ^nd conclusions 
fit with extant data and theories. Comments that reflect an understanding 
of the impact of disagreement on the validity of conclusions in also 
included under this category. Comments that are too general to be 
classified in the subordinate categories should be classified as 1 .7 
(Consensual Processes). On the other hand, comments about replication 
by the original researchers should be coded as 1.7.1 (Replication by the 
Original Researchers), and comments about independent replications 
should be coded as 1 .7.2 (Independent Replication). If the participant does 
not make clear reference to who has done the replication, the comment 
should be coded as 1.7. 

Examples 

How many studies have been conducted in this area? If many studies 
have been done, each finding the same thing then one can conclude that 
this insecticide is harmful. (P) [Note: Because "finding the 
same thing" could refer to replication (1.7.1 or 1.7.2) or fit with other 
evidence (1.7.3), this comment should be coded as 1.7.] 

What kind of studies have been done? The answer would show me if 

there was any supporting evidence. (P) [Note: The reference 

to "any supporting evidence" could be to replication (1.7.1 or 

1 .7.2) or to other kinds of evidence (1 .7.3). Because of the ambiguity, 

the comment should be coded as 1.7.] 
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1.7.1 Replication by the Original Researchers 

Comments about the value of replication by the original researchers 
(i.e.> the researchers involved in the study described in the report) are 
included here. Such comments are interpreted as reflecting 
knowledge that "good" scientists try to replicate their results. 

Example 

Have the researchers replicated their findings? If they did the same 
study again and found the same results, I would be more willing to 
accept their conclusion that the pesticide caused the decreased 
mating. 



1.7.2 Independent Replication 

Comments about the value of replication by independent researchers 
(i.e., not the original researchers) are included here. Such comments 
are interpreted as reflecting knowledge that an important activity of 
the scientific community is determining whether critical results can be 
replicated by independent researchers. 

Example 

Have other scientists around the world done the experiment and 
found the same results? If other scientists have found the same 
results then one knows that this gold producing process truly works 
and is no hoax. (P) 



1.7.3 Fit with other Evidence or Theories 

Included are comments reflecting knowledge that findings or 
conclusions are more compelling to the extent that they agree with 
extant data, concepts, and theories. The reference here is to results 
or conclusions from studies other than the one in the report. 
Comments regarding converging evidence obtained within the original 
study should be described in terms of validity (1.5.2). 



1.7.4 Consensual Processes - Other 

Included here are references to other specific issues concerning 
consensual processes that do not fit into categories 1.7.1 to 1.7.3. 



1.8 BOUNDARY CONDITIONS 
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Boundary conditions refer to the conditions under which a finding or 
conclusion are likely to be valid. Questions about boundary conditions 
reflect a sensitivity to the fact that findings or conclusions often are 
subject to limitations that might have important implications for the 
generality of conclusions. Thus, requests about the generalizability 
and/or specificity of effects are included under this category. Questions 
about the representativeness of the sample (i.e., whether the sample is 
sufficiently representative of the larger population to permit acceptance 
of the conclusion) should be included here as well. 

Examples 

Does this insecticide have effects on other animals? If it had effects on 
more animals, I could believe it to a greater extent. (P) 

Are the bodily procedures of a pregnant woman the same as in rats? 
Findings in rats does not mean it will happen in humans. (P) 

Was this done in the wild or in a lab? It would convince me more if the 
robins were affected like this in their natural habitat. (P) 



OTHER 

Included are other comments relevant to knowledge about good scientific 
practices and conventions that do not fit into the categories above. 
[Comments that fit more than one category should be coded as 1 .0 
(Knowledge About "Good" Scientific Practices and Conventions).] 
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20 KNOWLEDGE ABOUT THE SOCIAL CONTEXT OF SCIENCE 

Science exists in a social context, and many comments show an awareness of 
the potential influence of social factors on the quality of research and the 
validity of the data or conclusions. Characteristics of the researchers, the 
funding agencies, and the communication outlets (e.g., journals, conferences) 
often are viewed as being important indicators of good research. 

2.1 RESEARCHERS OR RESEARCH GROUPS 

Examples 

Who are the researchers? To find out if the data collected are correct. 
(P) 

Who are the researchers? To discover the validity of their data. (P) 

Who did the research? This would help me to understand if there is any 
substance behind the claim. (P) 



2.1.1 Quality 

Included are comments about the qualifications, training, reputation, 
and knowledge of the researcher and/or the prestige and quality of 
the research environment in which the researcher works. 

Examples 

Have professional researchers conducted the research in this area? I 
would be more willing to accept it if this is true. (P) 

Who did the research? If the research was done by a reputable 
organization I would be inclined to believe it. (P) 



2.1.2 Biases 

Included are comments about possible motivations or expectations on 
the part of the researcher that might bias the research. These biases 
may be intentional or unintentional, and they may originate within the 
researcher or from external sources. [Comments about biases 
associated with funding agencies should be coded as 2.2.2 (Biases).] 
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Example 

Were the researchers members of Autos for the Futures? Motivation 
for finding. (P) 



2.1.3 Researchers or Research Groups - Other 

Included here are references to other specific issues concerning 
the researchers or research groups that do not fit into categories 
2.1.1 or 2.1.2. 



2.2 SOURCE OF FUNDING 



2.2.1 Quality 

Included are comments about the reputation and prestige of the 
agency funding the research. 

Example 

Who funded this? If it was a reputable agency, I would be more 
willing to believe that the study was peer-reviewed and thus the data 
more believable. 



2.2.2 Biases 

Included are comments about possible motivations or expectations on 
the part of the funding agency that might bias the research. 

Example 

Who funded this? If the manufacturers of Quipmanol funded this, I 
would not be willing to believe their claim that it reduces 
pollution. 



2.2.3 Source of Funding - Other 

Included here are references to other specific issues concerning 
the source of funding that do not fit into categories 
2.2.1 or 2.2.2. 



2.3 COMMUNICATION OUTLET 
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2.3.1 Quality 

Included are comments about the reputation and prestige of the outlet 
in which the research was publicized, such as journals, books, 
conferences, newspapers, and radio interviews. 

Example 

Where was this research published? If it made it through a peer- 
reviewed journal, I would be more willing to believe the results. 



2.3.2 Biases 

Included are comments about possible motivations or expectations 
associated with the communication outlet that might bias the research 
or the information that is reported. 

Example 

Were these research findings taken from a pamphlet distributed by the 
makers of Quipmanol or from an unbiased source? A biased source may 
not present a true picture of the findings. 



2.3.3 Communication Outlet - Other 

Included here are references to other specific issues concerning 
the communication outlet that do not fit into categories 
2.3.1 or 2.3.2. 



2.4 PROMOTERS AND DETRACTORS 

Promoters and detractors include individuals or special interest groups 
who interpret and publicize scientific research to manipulate opinions or 
behavior. References to regular publications (e.g., newspapers) should be 
coded as 2.4. 



2.4.1 Quality 

Included are comments about the reputation and prestige of the 
promoters or detractors. 

Example 

Who are Mind Matters? If they are quacks, I won't believe the 
conclusion. 
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2.4.2 Biases 

Included are comments about possible motivations or expectations 
associated with the promoters or detractors that might bias the way 
in which the research is interpreted and/or used. 

Example 

Do Autos for the Future have a vested interest in Quipmanol? If 
they are out to make a buck, I won't believe the conclusion. 



2.4.3 Promoters and Detractors - Other 

Included here are references to other specific issues concerning 
the promoters and detractors that do not fit into categories 
2.4.1 or 2.4.2. 



2.5 EXPERT OPINION 

People often refer to the need for expert opinion in eva'uating scientific 
research, indicating knowledge that experts might have insights or 
criticisms that can be influential. Included are requests for comments of 
the original researchers or any other expert. Requests for the opinions of 
participants in the reported study should be treated as requests for 
additional measures, 1.5.2 (Validity of Measurement). 

Example 

What do the researchers say about the conclusion? I would be more 
likely to believe the underlined conclusion if the people who did the study 
agree with it. 

2.6 OTHER 

Included are other comments relevant to knowledge about the social 
context of science that do not fit into the categories above. (Comments 
that fit more than one category should be coded as 2.0.) 
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3.0 KNOWLEDGE ABOUT THE FUNCTIONS OF SCIENCE IN SOCIETY 

People sometimes use their knowledge about relations between science and 
society to evaluate conclusions from scientific research, even though these 
relations should be irrelevant. Particularly salient are two potential benefits of 
research: (a) the utility of scientific results for application in society; and (b) 
the informational value of scientific results for enhancing education or future 
research. 



3.1 UTILITY FOR APPLICATION 

Included are comments about the feasibility or expense involved in 
implementing changes based on th3 scientific research, or the probability 
that such changes would be used and useful. For example, the comments 
may refer to costs or benefits of using the agent, or the dangerousness 
and/or safety of using the agent outside of the research environment. 

Examples 

How does this poisonous chemical affect the environment and humans? 
If the chemical reduces air pollution, it may cause serious side effects 
also for the environment. (P) 

Why or how is this chemical poisonous to us? It would help me decide 
on the potential health risks for all living creatures versus the 
environmental benefits gained. (P) 

Are there other harmful effects? If cost outweighs the benefit then what 
good is it? (P) 

What does Quipmanol become after being combusted? If the products of 
combustions are harmless, it will reduce air pollution. (P) 



3.2 INFORMATION VALUE 

Included are comments about the extent to which the research 
contributes to what is already known. 



3.3 FUNCTIONS OF SCIENCE IN SOCIETY - OTHER 

Included here are references to issues concerning the seriousness of the 
problem that the treatment (i.e. research findings) may address or may 
cause. 
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Examples 

Why is it important to dream about the future? If the reasons are 
trivial, I am not interested in the research. 

At what dosage of Permaldrin do robins stop mating? If it's only 
damaging at very high concentrations, then it's not an important problem. 
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4.0 OTHER 

Included are comments that appear to be from specific knowledge domains 
that are different from 1.0 to 3.0. 

Example 

What is Mai Handu? I would like to try it for myself. 



5.0 AMBIGUOUS 

Included here are comments that are related to more than one of the 
knowledge domains described above. 

6.0 NO IDENTIFIABLE KNOWLEDGE DOMAIN FOR REQUEST 

Included are: (a) cases in which no response was provided, (b) comments that 
are off-task, (c) simple requests for additional information, and (d) remarks that 
are so profoundly ambiguous that we cannot determine whether a knowledge 
domain they represent a knowledge domain. With respect to comments that 
are off-task (b), participants must explicitly express disbelief to receive a code 
of 6.0. If disbelief is not explicit, then the code is determined by the thrust of 
the justification. 

Examples 

What hard-core evidence is there that praises this? If evidence is convincing, 
then I would be more willing to accept it. (P) [Note: The "reason" 
given is tautological and provides no clue to why hard-core evidence 
would be convincing.] 

Does this work for everyone? If it does I would be very interested and willing 
to try it, but still would be skeptical. (P) [Note: This justification 
is off task.] 

Does the type of music listened to while exercising matter, or does it have to 
be that particular type? Can other types of music be played and get the same 
results? (P) [Note: The first question appears to be related to 
1.3 (Controlling for Confounding Variables/Elimination of Other Causes) but the 
justification response reveals no information about knowledge domain.] 

What were the other people listening to? The other subjects' music could 
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have influenced the study. (P) [Note: No "other subjects" were 
described in the study, so this comment is interpreted as off task.] 

How fit were the people to begin with? Validity of results? (P) 

[Note: This comment is too ambiguous to determine whether a knowledge 

domain is involved.] 



IDENTIFICATION OF REQUESTS AND JUSTIFICATIONS 

Recall that, ideally, each request and the corresponding justification should 
represent a single idea, however participants sometimes express two or more 
ideas in a single response. When this occurs, it is necessary to use a 
systematic procedure for partitioning the requests and justifications into 
separate idea units. The procedure used to partition a single response into 
separate idea units was developed in the study of prose comprehension and 
memory (e.g., Voss et. al., 1982). The accepted procedure is to divide a 
written protocol into units of text that represent a single, complete idea. 
Responses typically take a subject-predicate form and va;y in length and 
complexity. 

IDENTIFICATION OF IDEA UNITS 

The first step in coding the protocol is for the coder to read through the 
entire response and decide if one or more ideas are contained in the 
response. Assume that the participants are using each numbered space to 
reflect one idea. If two or more ideas sb-jm to be present, thev 
should be separated only if they can be given different codes. Otherwise 
consider the ideas as equivalent expressions of the same idea and assign a 
single code . Typically, if subsequent ideas are presented parenthetically, a 
single code should be given, but if the ideas are separated by an "and" they 
should be given two codes . Indicate the division of dissimilar ideas by 
placing slash marks in the written protocol where one idea end? and the next 
begins (see examples below). Failure to make these divisions when dissimilar 
ideas are presented may lead to unreliable coding in later stages. 

Examples of partitioning requests: 

What age group was tested? (The participant is expressing one idea: age of 
subjects.) 

What are the harmful effects of Quipmanol? (The participant is expressing one 
idea: side effects.) 

Have other scientists around the world done the experiment and found the 
same result? (The participant is addressing one idea: other researchers 
finding this result when they did the experiment.) 
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How does this poisonous chemical affect the environment and humans? (The 
participant is asking two things but both are equivalent examples of the 
same idea: what are side effects of the chemical?) 

What is Gowia music and what does it sound like? (The participant is asking 
about the agent and a more specific property of the agent, so this 
should be treated as one idea because both would be given the same 
topic code.) 

Who funded this - Greenpeace? Independent granting agency? (The participant 
is asking two separate questions, yet both questions represent the same 
idea with the second question being an equivalent example of another 
funding source.) 

How controlled was the experiment // and how would you set up an 

experiment find out if this side effect would be present in women? (The 
participant is expressing two ideas: experimental control and testing for 
generalizability.) 

What is the element Knardium // and why haven't we heard about it? (The 
participant is expressing two ideas: identification of Knardium and 
familiarity with Knardium.) 

Examples of partitioning justifications: 

If ! understand the theory behind the concept I would be more likely to 

believe it. (The participant expresses one idea: theory of experiment.) 

To determine the validity of the experiment. (The participant is expressing one 
idea: experimental validity.) 

The kind of student depends a lot on how smart they are etc. Generally kids 
of higher social status learn better and faster and are more interested in 
succeeding. If they only tested higher income kids the test would be 
invalid. (The participant is expressing one idea by elaborating on the 
example.) 

If they are well known and well respected, I would be more likely to believe 
them. (The participant is expressing two ideas: familiarity of researchers 
and credibility of researchers, but both are equivalent expressions 
addressing the quality of the researchers. Thus only one code is given.) 

If a lot of people have been tested // and the findings are all positive, then I 
would believe it. (The participant is addressing two ideas: number of 
subjects tested and converging evidence. Although both of these ideas 
may relate to replicabilty of the results, they are not equivalent and an 
assumption would have to be made to connect the ideas as one; so this 
should be given two codes.) 

If research was done hy reputable people // using scientific methods I would 
have more faith in the results. (The participant is expressing two ideas: 
reputation of the researchers and method of investigation.) 

From the experiment conducted one can conclude whether biases were 

involved // or if it was conducted in a scientific manner. (The participant 
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is expressing two ideas: experimental bias and method of investigation.) 
NUMBERING THE RESPONSES 

The final step in coding might be to assign an identification number. In our 
research, a single, five digit number was assigned to each request and 
corresponding justification. The number identified the participant who the 
generated the response, the news brief the participant was reading, and the 
sequential order of the response. The first three digits refer to the 
participant number. The next digit represents the story, for example: 
1 = insecticide, 2 = dream crystals, 3 = meditation, 4 = poisonous chemical, 
5=chemical element, 6 = slow speed music, 7 = anti-nausea drug, or 
8=textbook (see Appendix 1). The last digit represents the ordinal 
position of the request and related justification (e.g.. 1 =first response 
generated; 4 = fourth response generated). 

For example, if the item being coded was the third response generated about 
the pesticide story for participant 123, the code which identifies this 
response would be 12313 (participant 123, first story, third response). In 
instances where two or more idea units are present in one response add an 
alphabetical extension to the coding, i.e., 12313a for the first idea, and 
12313b for the second. 

ELABORATED PROCEDURE FOR CODING 

Based on the procedures and guidelines for coding requests and justifications 
outlined in the previous sections, coding proceeds as follows: 

1) Read the response to the first prompt. 

2) Identify ideas that are requests and assign topic codes. 

3) If a justification is included in the request, assign a knowledge 

domain code. 

4) Read the response to the second prompt. 

5) Identify ideas that are justifications and assign knowledge 

domain codes. 

6) If a request is included in the justification, assign a topic code. 

7) Number the responses. 

Once you have used this procedure to score all your protocols, try to resume 
normal functioning by requesting a shore leave on the planet Risa (see Star 
Trek : The Next Generation). 
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APPENDIX 1 

SAMPLE STORIES AND QUESTIONNAIRE PROMPTS 
INSECTICIDE (1) 

People are concerned that declines in wildlife populations will result in 
extinction for some species. Researchers have reported that robins that have 
been exposed to the insecticide Permaldrin are less likely to mate than 
usual. Members of Nature Unlimited have hailed this finding and have 
concluded that using this insecticide is an impo rtant factor in causing a 
decline in robins' mating behavior . 

In the news brief above, please pay special attention to the 
underlined conclusion as you respond to the following three items in 
Section 1 and the request in Section 2. 

SECTION 1 

How likely do you think it is that the underlined conclusion is true? 

Very 1 2 3 4 5 6 7 Very 
Unlikely Likely 

How much experience with or knowledge of the general topic of the 
underlined conclusion do you have? 

Little 1 2 3 4 5 6 7 Much 
Experience or Experience or 

Knowledge Knowledge 

How closely related is the underlined conclusion to topics covered in 
natural and physical sciences such as Physics, Chemistry, or Biology? 

Very 1 2 3 4 5 6 7 Very 
Unrelated Related 

How interested are you in the general topic of the underlined conclusion? 

Very 1 2 3 4 5 6 7 Very 
Uninterested Interested 
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SECTION 2 

Suppose that the underlined conclusion is very important to you and that 
you must determine whether it is true. Please generate a list of as many 
questions as you can that you would want to have answered before you 
decide whether the conclusion made by members of Nature Unlimited is true. 
Also, for each question you list, please indicate how you think the answer to 
that question would help you to evaluate the conclusion in the news brief. 

1 . What is the first question you would want answered? 

How would an answer to this question help you to decide whether 
the underlined conclusion in the news brief is true? 



2. What is the next question you would want answered? 

How would an answer to this question help you to decide whether 
the underlined conclusion in the news brief is true? 



DREAM CRYSTALS (2) 
People have long been interested in what the future holds. Researchers have 
reported that people who wear Ollinite crystals during sleep are more likely to 
have dreams that predict the future. Members of Mind Matters have hailed 
this finding and have concluded that wearing this crystal is important for 
increasing the frequency of dreams about fiiture events . 



MEDITATION (3) 

People in Western countries have long been fascinated by traditional Eastern 
religious practices. Researchers have reported that senior citizens who 
practice Mai Handu meditation show an increased sense of well being. 
Members of Lifestyles for Seniors have hailed this finding and have concluded 
that practicing this meditation is important for increasing the sense of well- 
being in seniors . 



POISONOUS CHEMICAL (4) 
People are concerned about the environmental effects of automobile 
emissions. Researchers have reported that vehicles that burn gasoline 
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containing the poisonous chemical Quipmanol will reduce existing levels of air 
pollution. Members of Autos for the Future have hailed this finding and have 
concluded that fu eling vehicles with a gasoline containing this poisonous 
chemical is important for decreasing current levels of air pollution. 



CHEMICAL ELEMENT (5) 
People have long been interested in possession of rare and precious metals. 
Researchers have reported that treating zinc with the element 
Knardium will produce gold. Members of Innovations in Chemistry have 
hailed this finding and have concluded that treating zinc with this chemical 
element is important for producing ooid . 



SLOW SPEED MUSIC (6) 
People have shown a great interest in physical fitness and body weight. 
Researchers have reported that people listening to slow speed Gowla music 
while exercising will burn more calories. Members of the Healthy Lifestyle 
Group have hailed this finding and have concluded that listening to this slow 
speed music while exercising is important for burning more calories . 



ANTI-NAUSEA DRUG (7) 
There has been increased public interest in the side effects of drugs. 
Researchers have reported that pregnant rats given the anti-nausea drug 
Alprozam are more likely to experience miscarriages. Members of the 
National Birth Institute have hailed this finding and have concluded that 
taking this anti-nausea drug during pregnancy is an important factor in 
causing miscarriages . 



TEXTBOOK (8) 

The academic performance of youth is of concern to educators, scientists, 
and parents. Researchers have reported that students in schools where 
the "Science for Life" textbook is used score higher on the provincial science 
tests. Members of Learning Now have hailed this finding and have concluded 
that using this textbook is important for improving high school students' 
scores on provincial science tests . 
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APPENDIX 2 
SUMMARY OF TOPIC CODES 

1.0 SOCIAL CONTEXT 

1.1 PEOPLE 

1.1.1 Researchers/Experimenters 

1.1.1.1 Identification 

1.1.1.2 Number 

1 .1 .1 .3 Testimonials 

1 .1 .1 .4 Training/Qualification/Credentials 

1.1.1.5 Motivation/Expectations/Biases/Beliefs 

1.1.1.6 Researchers/Experimenters - Other 

1.1.2 Promoters and Detractors 

1 .1 .2.1 Identification 

1.1.2.2 Number 

1 .1 .2.3 Testimonials 

1 .1 .2.4 Training/Qualifications/Credentials 

1 .1 .2.5 Motivation/Expectations/Biases 

1.1.2.6 Promoters and Detractors - Other 

1.1.3 Creators of Agent 

1 .1 .3.1 Identification 

1.1.3.2 Number 

1 .1 .3.3 Testimonials 

1 .1 .3.4 Training/Qualification/Credentials 

1.1.3.5 Motivation/Biases 

1.1.3.6 Creators of Agent - Other 

1.1.4 Other Experts 

1 .1 .4.1 Identification 

1.1.4.2 Number 

1 .1 .4.3 Testimonials 

1 .1 .4.4 Training/Qualification/Credentials 

1.1.4.5 Motivation/Biases 

1 .1 .4.6 Other Experts - Other 

1.1.5 People - Other 

1.2 SOURCE OF INFORMATION 

1.2.1 Publication Outlet 

1.2.2 Author(s) or Broadcasters 

1.2.3 Source of Information - Other 

1.3 FUNDING ISSUES 

1.4 LOCATION OF RESEARCH - IDENTIFICATION 

1.5 SOURCE OF RESEARCH QUESTION - IDENTIFICATION 

1.6 SOCIAL CONTEXT - OTHER 
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2.0 AGENT 

2.1 AGENT IDENTIFICATION/PROPERTIES 

2.2 AGENT MECHANISMS, EFFECTS/SIDE EFFECTS (THEORY) 

Agent Mechanisms 
Side Effects 

2.3 ALTERNATIVE AGENTS 

2.4 AGENT - OTHER 

3.0 METHODS 

3.1 DESIGN 

3.1.1 Type of Design 

3.1.2 Research Context 

3.1.3 Duration of the Research 

3.1.4 Control of Other Factors/Confounding Factors 
General 

Specific 

3.1.5 Design - OTHER 

3.2 AGENT DELIVERY (PROCEDURE) 

3.2.1 Nature of Agent Delivery 

3.2.2 Setting 

3.2.3 Calendar Time 

3.2.4 Agent Delivery - Other 

3.3 SUBJECTS 

3.3.1 Identification 

3.3.2 Qualitative Characteristics 
3.3.3. Number of Subjects Tested 
3.3.4 Sample Selection 

3.3.5. Representativeness of Sample 

3.3.6. Expectations/Motivations/Awareness/Beliefs 

3.3.7. Subjects - Other 

3.4 MEASURES 

3.4.1 Identification of Measures 
Basic 

Technical-Operational Definition of Dependent Variable 

3.4.2 Manipulation Check 

3.4.3 Evaluation of the measures 

3.4.4 Additional measures 

3.4.5 Measures - Other 

3.5 CONSISTENCY OF METHODS/PROCEDURES 

3.6 REPLICABILITY OF METHODS/PROCEDURES 

3.7 METHODS - OTHER 
4.0 DATA/STATISTICS 

4.1 DATA 

4.1.1 Absolute Nature of tha Data for the Dependent Variables 
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4.1.2 Comparative Nature of the Data for the Dependent Variable 

4.1.3 Duration of Effect 

4.1.4 Additional data 

4.1.5 Data - Other 
4.2 STATISTICS 

4.2.1 Type/Nature of Statistical Analysis 

4.2.2 Appropriateness of Statistics 

4.2.3 Statistics - Other 

5.0 RELEVANCE OF THE AGENT/RESEARCH ON THE AGENT 

5.1 G EN ERALIZ ABMJTY/SPECI Fl CITY OF AGENT EFFECT 

5.2 PRACTICALITY/UTILITY/FUNCTION OF AGENT APPLICATION 

5.3 RECENCY OF RESEARCH 

5.4 IMPACT OF THE AGENT/RESEARCH FINDINGS/CONCLUSION 

5.5 CONTRIBUTION OF THIS RESEARCH TO THE DOMAIN 

5.6 AUDIENCE FAMILIARITY WITH THE RESEARCH FINDINGS 

5.7 RELEVANCE - OTHER 



6.0 RELATED RESEARCH (BEYOND THE STUDY DESCRIBED) 

6.1 SIMILAR DOMAIN OF STUDY 

6.2 METHOD OF STUDY 

6.3 SUPPORTING DATA/DATA REPLICATION 

6.4 CONSENSUS/NONCONSENSUS 

6.5 RELATED RESEARCH - OTHER 



7.0 OTHER 



8.0 AMBIGUOUS BUT RELEVANT TO THE RESEARCH DESCRIBED 
9.0 OFFTASK (IRRELEVANT TO THE STUDY) 
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APPENDIX 3 

SUMMARY OF KNOWLEDGE DOMAIN CODES 

1.0 KNOWLEDGE ABOUT "GOOD" SCIENTIFIC PRACTICES AND 
CONVENTIONS 

1.1 ADEQUACY OF RESEARCH DESIGN 

1 .2 THEORETICAL EXPLANATION 

1.3 CONTROLLING FOR CONFOUNDING VARIABLES/ELIMINATION OF OTHER 

CAUSES 

1.3.1 Subjects 

1.3.2 Treatment 

1.3.3 Alternative Explanations - Other 

1.4 IDENTIFICATION OF AGENTS OR TREATMENTS WITH IDENTICAL EFFECTS 

1.5 MEASUREMENT 

1.5.1 Reliability 

1.5.2 Validity 

1.5.3 Measurement - Other 

1.6 STATISTICAL INFERENCES 

1.6.1 Sample size 

1.6.2 Effect size 

1.6.3 Variability 

1.6.4 Statistical Significance and Chance 

1.6.5 Statistical Inferences - Other 

1.7 CONSENSUAL PROCESSES 

1.7.1 Replication by the Original Researchers 

1.7.2 Independent Replication 

1.7.3 Fit with other Evidence or Theories 

1.7.4 Consensual Processes - Other 

1.8 BOUNDARY CONDITIONS 

1.9 OTHER 

2.0 KNOWLEDGE ABOUT THE SOCIAL CONTEXT OF SCIENCE 

2.1 RESEARCHERS OR RESEARCH GROUPS 

2.1.1 Quality 

2.1.2 Biases 

2.1.3 Researchers - Other 

2.2 SOURCE OF FUNDING 

2.2.1 Quality 

2.2.2 Biases 

2.2.3 Source of Funding - Other 

2.3 COMMUNICATION OUTLET 

2.3.1 Quality 

2.3.2 Biases 
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2.3.3 Communication Outlet - Other 

2.4 PROMOTERS AND DETRACTORS 

2.4.1 Quality 

2.4.2 Biases 

2.4.3 Promoters and Detractors - Other 

2.5 EXPERT OPINION 

2.6 OTHER 

3.0 KNOWLEDGE ABOUT THE FUNCTIONS OF SCIENCE IN SOCIETY 

3.1 UTILITY FOR APPLICATION 

3.2 INFORMATION VALUE 

3.3 OTHER 

4.0 OTHER 

5.0 AMBIGUOUS 

6.0 NO IDENTIFIABLE KNOWLEDGE DOMAIN 
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single or multiple, 72 
Strict and lenient criteria, see under Coding 
Topics 

assignment of, 6-8 

definition, 6 
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